[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:05 dist-test-slave-df53 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:05 dist-test-slave-df53 krb5kdc[395](info): set up 2 sockets
Sep 27 22:51:05 dist-test-slave-df53 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 22:51:06.426106 388 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:06 dist-test-slave-df53 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:06.589784 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43767
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:06.904404 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:06.904685 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:06.904768 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:06.911752 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:06.911851 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:06.911896 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:06.911947 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:06.911990 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:06.921331 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:06.922573 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:06.923883 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:06.935911 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:06.935885 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:06.935930 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:06.936661 408 server_base.cc:1034] running on GCE node
I20260927 22:51:06.937323 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:06.939085 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:06.940352 408 hybrid_clock.cc:648] HybridClock initialized: now 1790549466940276 us; error 67 us; skew 500 ppm
I20260927 22:51:06.940631 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:06 dist-test-slave-df53 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:06.943509 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:06.944957 408 webserver.cc:460] Webserver started at http://127.0.97.62:40871/ using document root <none> and password file <none>
I20260927 22:51:06.945657 408 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:06.945788 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:06.946069 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:06.948851 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "819f67e31c114dfeb0187209407bf006"
format_stamp: "Formatted at 2026-09-27 22:51:06 on dist-test-slave-df53"
server_key: "79ae94e4589d7ca77be4a71cdd0dff40"
server_key_iv: "9f2ac4af18b6fb4a85d5caa581da3158"
server_key_version: "encryptionkey@0"
I20260927 22:51:06.949568 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "819f67e31c114dfeb0187209407bf006"
format_stamp: "Formatted at 2026-09-27 22:51:06 on dist-test-slave-df53"
server_key: "79ae94e4589d7ca77be4a71cdd0dff40"
server_key_iv: "9f2ac4af18b6fb4a85d5caa581da3158"
server_key_version: "encryptionkey@0"
I20260927 22:51:06.954577 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:51:06.957824 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:06.959048 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:51:06.959229 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "819f67e31c114dfeb0187209407bf006"
format_stamp: "Formatted at 2026-09-27 22:51:06 on dist-test-slave-df53"
server_key: "79ae94e4589d7ca77be4a71cdd0dff40"
server_key_iv: "9f2ac4af18b6fb4a85d5caa581da3158"
server_key_version: "encryptionkey@0"
I20260927 22:51:06.959414 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:06.983055 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:06.986868 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:06.987160 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:07.006108 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43767
I20260927 22:51:07.006129 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43767 every 8 connection(s)
I20260927 22:51:07.007407 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:07.010993 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:07.014891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 408
I20260927 22:51:07.015189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:07.017433 388 external_mini_cluster.cc:1428] Setting key 5384bece72b7568d51ce8d36f727d56a
I20260927 22:51:07.023576 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006: Bootstrap starting.
I20260927 22:51:07.027351 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:07.028573 475 log.cc:824] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:07.031913 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006: No bootstrap required, opened a new log
I20260927 22:51:07.042601 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819f67e31c114dfeb0187209407bf006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43767 } }
I20260927 22:51:07.042932 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:07.043059 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f67e31c114dfeb0187209407bf006, State: Initialized, Role: FOLLOWER
I20260927 22:51:07.043718 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819f67e31c114dfeb0187209407bf006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43767 } }
I20260927 22:51:07.044409 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:07.044561 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:07.044724 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:07.047458 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819f67e31c114dfeb0187209407bf006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43767 } }
I20260927 22:51:07.048022 475 leader_election.cc:302] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 819f67e31c114dfeb0187209407bf006; no voters:
I20260927 22:51:07.048651 475 leader_election.cc:288] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:07.048767 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:07.050182 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [term 1 LEADER]: Becoming Leader. State: Replica: 819f67e31c114dfeb0187209407bf006, State: Running, Role: LEADER
I20260927 22:51:07.050835 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:07.050691 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819f67e31c114dfeb0187209407bf006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43767 } }
Sep 27 22:51:07 dist-test-slave-df53 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:07.055064 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "819f67e31c114dfeb0187209407bf006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819f67e31c114dfeb0187209407bf006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43767 } } }
I20260927 22:51:07.055212 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 819f67e31c114dfeb0187209407bf006. Latest consensus state: current_term: 1 leader_uuid: "819f67e31c114dfeb0187209407bf006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819f67e31c114dfeb0187209407bf006" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43767 } } }
I20260927 22:51:07.055855 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:07.055852 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:07.063968 488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:07.066987 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:07.021820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.0.97.62:43767)
0927 22:51:07.022647 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:07.022674 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:07.024758 (+ 2084us) server_negotiation.cc:407] Connection header received
0927 22:51:07.037297 (+ 12539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:07.037315 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:07.037720 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:07.038061 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:07.041669 (+ 3608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:07.042985 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:07.045811 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:07.046455 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:07.046621 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:07.054266 (+ 7645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:07.054280 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:07.054301 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:07.054373 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:07.060591 (+ 6218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:07.061903 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:07.061915 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:07.061927 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:07.062196 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:07.062917 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:07.062927 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:07.062933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:07.063761 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:07.064321 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:07.064939 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 22:51:07.065337 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":390,"threads_started":1}
I20260927 22:51:07.072831 488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:07.082197 488 catalog_manager.cc:1347] Generated new cluster ID: a34e9a6c4081467386ac7bcfc46ecca3
I20260927 22:51:07.082315 488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:07.102129 488 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:07.103881 488 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:07.111550 488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 819f67e31c114dfeb0187209407bf006: Generated new TSK 0
I20260927 22:51:07.112531 488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:07.156529 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43767
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:07.475487 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:07.475761 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:07.475945 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:07.483467 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:07.483641 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:07.483847 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:07.493525 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:07.494771 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:07.496037 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:07.509251 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:07.509222 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:07.509660 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:07.510041 501 server_base.cc:1034] running on GCE node
I20260927 22:51:07.510591 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:07.511449 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:07.512701 501 hybrid_clock.cc:648] HybridClock initialized: now 1790549467512646 us; error 57 us; skew 500 ppm
I20260927 22:51:07.513015 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:07 dist-test-slave-df53 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:07.519024 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:07.520818 501 webserver.cc:460] Webserver started at http://127.0.97.1:38493/ using document root <none> and password file <none>
I20260927 22:51:07.521463 501 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:07.521662 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:07.521957 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:07.524654 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "839bbea2e7334974b132e941bdbda6ac"
format_stamp: "Formatted at 2026-09-27 22:51:07 on dist-test-slave-df53"
server_key: "fb6fe560e5580a7a6a3d78245d6123c4"
server_key_iv: "1cdf89fcf4117d57784eaaea9399d316"
server_key_version: "encryptionkey@0"
I20260927 22:51:07.525388 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "839bbea2e7334974b132e941bdbda6ac"
format_stamp: "Formatted at 2026-09-27 22:51:07 on dist-test-slave-df53"
server_key: "fb6fe560e5580a7a6a3d78245d6123c4"
server_key_iv: "1cdf89fcf4117d57784eaaea9399d316"
server_key_version: "encryptionkey@0"
I20260927 22:51:07.530110 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:51:07.533118 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:07.534343 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:51:07.534502 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "839bbea2e7334974b132e941bdbda6ac"
format_stamp: "Formatted at 2026-09-27 22:51:07 on dist-test-slave-df53"
server_key: "fb6fe560e5580a7a6a3d78245d6123c4"
server_key_iv: "1cdf89fcf4117d57784eaaea9399d316"
server_key_version: "encryptionkey@0"
I20260927 22:51:07.534686 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:07.559548 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:07.564051 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:07.564344 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:07.565258 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:07.566609 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:07.566713 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:07.566854 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:07.566922 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:07.607432 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43775
I20260927 22:51:07.607476 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43775 every 8 connection(s)
I20260927 22:51:07.608687 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:07.611932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 501
I20260927 22:51:07.612164 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:07.612804 388 external_mini_cluster.cc:1428] Setting key d145cf4acf7220504017520e774b09ee
Sep 27 22:51:07 dist-test-slave-df53 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:51:07.630156 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:07.611150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37603 (local address 127.0.97.62:43767)
0927 22:51:07.611712 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:07.611720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:07.612650 (+ 930us) server_negotiation.cc:407] Connection header received
0927 22:51:07.615022 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:07.615028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:07.615110 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:07.615232 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:07.616924 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:07.618099 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:07.619656 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:07.620054 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:07.620189 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:07.624398 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:07.624403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:07.624406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:07.624434 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:07.627025 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:07.627893 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:07.627897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:07.627899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:07.628005 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:07.628519 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:07.628523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:07.628525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:07.628777 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:07.628894 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:07.629743 (+ 849us) server_negotiation.cc:299] Negotiation successful
0927 22:51:07.629893 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
I20260927 22:51:07.631049 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:07.611707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43767 (local address 127.0.97.1:37603)
0927 22:51:07.612474 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:07.612518 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:07.614784 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:07.615409 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:07.615417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:07.615884 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:07.616409 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:07.616425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:07.618285 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:07.618290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:07.619489 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:07.619509 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:07.620269 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:07.620277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:07.620600 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:07.621286 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:07.621318 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:07.624189 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:07.627170 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:07.627178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:07.627193 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:07.627778 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:07.628116 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:07.628121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:07.628125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:07.628338 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:07.628925 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:07.628933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:07.629330 (+ 397us) client_negotiation.cc:769] Sending connection context
0927 22:51:07.629680 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 22:51:07.630023 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":344,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:51:07.632647 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43767
I20260927 22:51:07.632954 629 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:07.633780 629 heartbeater.cc:511] Master 127.0.97.62:43767 requested a full tablet report, sending...
I20260927 22:51:07.636317 440 ts_manager.cc:194] Registered new tserver with Master: 839bbea2e7334974b132e941bdbda6ac (127.0.97.1:43775)
I20260927 22:51:07.638566 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:07.652575 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43767
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:07.960383 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:07.960664 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:07.960776 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:07.968133 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:07.968228 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:07.968336 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:07.977548 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:07.978716 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:07.979938 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:07.992609 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:07.992700 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:07.992717 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:07.993239 635 server_base.cc:1034] running on GCE node
I20260927 22:51:07.993976 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:07.994832 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:07.996078 635 hybrid_clock.cc:648] HybridClock initialized: now 1790549467996037 us; error 73 us; skew 500 ppm
I20260927 22:51:07.996376 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:07 dist-test-slave-df53 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:07.999473 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:08.001137 635 webserver.cc:460] Webserver started at http://127.0.97.2:43273/ using document root <none> and password file <none>
I20260927 22:51:08.001825 635 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:08.001919 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:08.002161 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:08.004982 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "a1e9d52b19ea457b82135ff0017c582c"
format_stamp: "Formatted at 2026-09-27 22:51:07 on dist-test-slave-df53"
server_key: "c83f63ee3f5099c75b64ea295aef609f"
server_key_iv: "683771ecc69a2ffc9da0b07c8663fd22"
server_key_version: "encryptionkey@0"
I20260927 22:51:08.005973 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a1e9d52b19ea457b82135ff0017c582c"
format_stamp: "Formatted at 2026-09-27 22:51:07 on dist-test-slave-df53"
server_key: "c83f63ee3f5099c75b64ea295aef609f"
server_key_iv: "683771ecc69a2ffc9da0b07c8663fd22"
server_key_version: "encryptionkey@0"
I20260927 22:51:08.011983 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 22:51:08.015983 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:08.017397 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:51:08.017535 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "a1e9d52b19ea457b82135ff0017c582c"
format_stamp: "Formatted at 2026-09-27 22:51:07 on dist-test-slave-df53"
server_key: "c83f63ee3f5099c75b64ea295aef609f"
server_key_iv: "683771ecc69a2ffc9da0b07c8663fd22"
server_key_version: "encryptionkey@0"
I20260927 22:51:08.017784 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:08.034864 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:08.039578 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:08.039925 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:08.040869 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:08.042651 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:08.042750 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:08.042888 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:08.042963 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:08.074914 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40129
I20260927 22:51:08.074946 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40129 every 8 connection(s)
I20260927 22:51:08.076107 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:08.077101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 635
I20260927 22:51:08.077293 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:08.077853 388 external_mini_cluster.cc:1428] Setting key e21549c4157ab3ed714ec00370c54ab5
Sep 27 22:51:08 dist-test-slave-df53 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:51:08.099684 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:08.079045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39317 (local address 127.0.97.62:43767)
0927 22:51:08.079225 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:08.079231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:08.080889 (+ 1658us) server_negotiation.cc:407] Connection header received
0927 22:51:08.082703 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:08.082708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:08.082786 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:08.082905 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:08.085059 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.085890 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.088762 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.089159 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.089253 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:08.093514 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:08.093519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:08.093523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:08.093554 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:08.096679 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.097355 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.097362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.097366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.097537 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.098015 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.098020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.098025 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.098340 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:08.098480 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:08.099123 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 22:51:08.099329 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:08.100629 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:08.079577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43767 (local address 127.0.97.2:39317)
0927 22:51:08.080756 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:08.080792 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:08.082446 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:08.083175 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:08.083189 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:08.084341 (+ 1152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:08.084888 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:08.084905 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.086043 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.086053 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:08.088590 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:08.088610 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.089397 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.089408 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:08.089807 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:08.090554 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:08.090592 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:08.093310 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:08.096821 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:08.096830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:08.096845 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:08.097238 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:08.097705 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:08.097709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:08.097711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:08.097895 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:08.098472 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:08.098480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:08.098794 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 22:51:08.099046 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:51:08.099531 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"thread_start_us":765,"threads_started":1}
I20260927 22:51:08.102262 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43767
I20260927 22:51:08.102571 763 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:08.103281 763 heartbeater.cc:511] Master 127.0.97.62:43767 requested a full tablet report, sending...
I20260927 22:51:08.104849 440 ts_manager.cc:194] Registered new tserver with Master: a1e9d52b19ea457b82135ff0017c582c (127.0.97.2:40129)
I20260927 22:51:08.105669 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:08.120260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43767
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:08.431734 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:08.431996 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:08.432184 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:08.439321 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:08.439421 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:08.439631 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:08.449203 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:08.450470 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:08.451717 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:08.464790 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:08.464913 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:08.465317 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:08.465867 768 server_base.cc:1034] running on GCE node
I20260927 22:51:08.466393 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:08.467244 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:08.468463 768 hybrid_clock.cc:648] HybridClock initialized: now 1790549468468423 us; error 30 us; skew 500 ppm
I20260927 22:51:08.468786 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:08 dist-test-slave-df53 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:08.471812 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:08.473193 768 webserver.cc:460] Webserver started at http://127.0.97.3:34683/ using document root <none> and password file <none>
I20260927 22:51:08.473891 768 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:08.474032 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:08.474321 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:08.477026 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "f1b9eeef5efb422ebd36b5bf038e4c47"
format_stamp: "Formatted at 2026-09-27 22:51:08 on dist-test-slave-df53"
server_key: "9a8c2dfbbf6e7e8ef9d33de94d3e9d8e"
server_key_iv: "837d98cb592fc916c4428a53cc0fcd4f"
server_key_version: "encryptionkey@0"
I20260927 22:51:08.477823 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f1b9eeef5efb422ebd36b5bf038e4c47"
format_stamp: "Formatted at 2026-09-27 22:51:08 on dist-test-slave-df53"
server_key: "9a8c2dfbbf6e7e8ef9d33de94d3e9d8e"
server_key_iv: "837d98cb592fc916c4428a53cc0fcd4f"
server_key_version: "encryptionkey@0"
I20260927 22:51:08.482647 768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:51:08.486608 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:08.487903 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:51:08.488088 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "f1b9eeef5efb422ebd36b5bf038e4c47"
format_stamp: "Formatted at 2026-09-27 22:51:08 on dist-test-slave-df53"
server_key: "9a8c2dfbbf6e7e8ef9d33de94d3e9d8e"
server_key_iv: "837d98cb592fc916c4428a53cc0fcd4f"
server_key_version: "encryptionkey@0"
I20260927 22:51:08.488296 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:08.506832 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:08.510640 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:08.510921 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:08.511814 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:08.513118 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:08.513222 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:08.513367 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:08.513445 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:08.548285 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44189
I20260927 22:51:08.548336 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44189 every 8 connection(s)
I20260927 22:51:08.549638 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:08.554529 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 768
I20260927 22:51:08.554713 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:08.555274 388 external_mini_cluster.cc:1428] Setting key b0a607d1954454a4d3f917c36714b7a4
Sep 27 22:51:08 dist-test-slave-df53 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:08.570106 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:08.552327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36195 (local address 127.0.97.62:43767)
0927 22:51:08.552529 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:08.552534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:08.553730 (+ 1196us) server_negotiation.cc:407] Connection header received
0927 22:51:08.555982 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:08.555993 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:08.556100 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:08.556209 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:08.558126 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.559185 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.560633 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.561016 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.561109 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:08.564729 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:08.564734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:08.564737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:08.564767 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:08.567356 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.567995 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.567999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.568001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.568109 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.568476 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.568479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.568481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.568738 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:08.568844 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:08.569497 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 22:51:08.569731 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:51:08.571125 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:08.552802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43767 (local address 127.0.97.3:36195)
0927 22:51:08.553539 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:08.553575 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:08.555662 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:08.556474 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:08.556485 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:08.557199 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:08.557970 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:08.557989 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.559319 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.559323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:08.560501 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:08.560512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.561191 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.561196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:08.561373 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:08.562025 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:08.562050 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:08.564537 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:08.567465 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:08.567472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:08.567482 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:08.567875 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:08.568199 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:08.568202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:08.568204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:08.568359 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:08.568842 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:08.568847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:08.569130 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 22:51:08.569397 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 22:51:08.569889 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":340,"threads_started":1}
I20260927 22:51:08.572408 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43767
I20260927 22:51:08.572664 896 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:08.573261 896 heartbeater.cc:511] Master 127.0.97.62:43767 requested a full tablet report, sending...
I20260927 22:51:08.574654 440 ts_manager.cc:194] Registered new tserver with Master: f1b9eeef5efb422ebd36b5bf038e4c47 (127.0.97.3:44189)
I20260927 22:51:08.575464 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36195
I20260927 22:51:08.583344 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:51:08 dist-test-slave-df53 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549468, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:51:08 dist-test-slave-df53 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549468, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:08.626665 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:08.611646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.0.97.1:43775)
0927 22:51:08.612453 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:08.612477 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:08.612502 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:51:08.612613 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:08.612618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:08.612798 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:08.612917 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:08.614348 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.615344 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.616589 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.617120 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.617255 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:08.621210 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:08.621220 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:08.621233 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:08.621261 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:08.623963 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.624618 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.624625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.624627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.624750 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.625145 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.625151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.625155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.625465 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:08.625778 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:08.626062 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 22:51:08.626304 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":573,"threads_started":1}
W20260927 22:51:08.627813 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56366
I20260927 22:51:08.641230 629 heartbeater.cc:503] Master 127.0.97.62:43767 was elected leader, sending a full tablet report...
Sep 27 22:51:08 dist-test-slave-df53 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:51:08 dist-test-slave-df53 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:08.668409 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:08.655228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.0.97.1:43775)
0927 22:51:08.655441 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:08.655454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:08.655594 (+ 140us) server_negotiation.cc:407] Connection header received
0927 22:51:08.655728 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:08.655732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:08.655815 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:08.655895 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:08.657211 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.657993 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.659468 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.659852 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.659942 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:08.663647 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:08.663651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:08.663655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:08.663683 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:08.666311 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.666880 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.666883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.666885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.666991 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.667284 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.667287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.667289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.667543 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:08.667704 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:08.667929 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 22:51:08.668073 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:51:08.669950 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 501
I20260927 22:51:08.682027 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:08.670852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.0.97.1:43775)
0927 22:51:08.671057 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:08.671063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:08.671279 (+ 216us) server_negotiation.cc:407] Connection header received
0927 22:51:08.671373 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:08.671377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:08.671440 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:08.671515 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:08.672551 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.673217 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.674497 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.674919 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.675020 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:08.677005 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:08.677009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:08.677013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:08.677039 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:08.679954 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.680481 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.680485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.680487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.680601 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.680885 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.680888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.680890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.681099 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:08.681272 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:08.681472 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 22:51:08.681672 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:51:08.682716 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:08.763747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 501
I20260927 22:51:08.779524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 635
Sep 27 22:51:08 dist-test-slave-df53 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:08.794842 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:08.780340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.0.97.2:40129)
0927 22:51:08.780822 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:08.780829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:08.780853 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:51:08.780909 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:08.780915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:08.781120 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:08.781292 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:08.782532 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.783574 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.784793 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.785317 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.785449 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:08.789698 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:08.789707 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:08.789721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:08.789752 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:08.792373 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.793015 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.793022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.793024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.793134 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.793448 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.793451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.793479 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.793852 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:08.794066 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:08.794332 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 22:51:08.794540 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":255,"threads_started":1}
I20260927 22:51:08.796128 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:08.875361 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 635
I20260927 22:51:08.896802 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 768
Sep 27 22:51:08 dist-test-slave-df53 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:51:08.912899 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:08.898073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.0.97.3:44189)
0927 22:51:08.898748 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:08.898754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:08.898784 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:51:08.898843 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:08.898849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:08.899054 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:08.899189 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:08.900363 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.901407 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.902737 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:08.903296 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:08.903411 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:08.907525 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:08.907536 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:08.907550 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:08.907581 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:08.910224 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.910875 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.910882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.910884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.910999 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:08.911365 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:08.911373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:08.911376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:08.911695 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:08.911998 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:08.912305 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:51:08.912567 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":391,"threads_started":1}
I20260927 22:51:08.914172 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:08.994719 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 768
I20260927 22:51:09.010785 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 408
I20260927 22:51:09.011945 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:09.110190 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 408
2026-09-27T22:51:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3774 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:09 dist-test-slave-df53 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:09 dist-test-slave-df53 krb5kdc[917](info): set up 2 sockets
Sep 27 22:51:09 dist-test-slave-df53 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 22:51:10.185379 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:10 dist-test-slave-df53 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:10.349080 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39657
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:10.659148 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:10.659466 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:10.659613 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:10.666772 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:10.666903 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:10.667002 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:10.667104 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:10.667196 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:10.676666 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:10.677943 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:10.679148 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:10.690331 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:10.690452 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:10.690483 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:10.691452 930 server_base.cc:1034] running on GCE node
I20260927 22:51:10.692148 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:10.693392 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:10.694613 930 hybrid_clock.cc:648] HybridClock initialized: now 1790549470694542 us; error 84 us; skew 500 ppm
I20260927 22:51:10.694931 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:10 dist-test-slave-df53 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:10.697922 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:10.699218 930 webserver.cc:460] Webserver started at http://127.0.97.62:41913/ using document root <none> and password file <none>
I20260927 22:51:10.699802 930 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:10.699893 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:10.700150 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:10.703164 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "a60ab767f0954afd88109b1f7c407313"
format_stamp: "Formatted at 2026-09-27 22:51:10 on dist-test-slave-df53"
server_key: "75df9bcc6917ce617845af10fe23b0a1"
server_key_iv: "bb794c8c2075c63ddc6abdb7f06f56df"
server_key_version: "encryptionkey@0"
I20260927 22:51:10.703856 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "a60ab767f0954afd88109b1f7c407313"
format_stamp: "Formatted at 2026-09-27 22:51:10 on dist-test-slave-df53"
server_key: "75df9bcc6917ce617845af10fe23b0a1"
server_key_iv: "bb794c8c2075c63ddc6abdb7f06f56df"
server_key_version: "encryptionkey@0"
I20260927 22:51:10.708690 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:51:10.712000 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:10.713255 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:51:10.713421 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "a60ab767f0954afd88109b1f7c407313"
format_stamp: "Formatted at 2026-09-27 22:51:10 on dist-test-slave-df53"
server_key: "75df9bcc6917ce617845af10fe23b0a1"
server_key_iv: "bb794c8c2075c63ddc6abdb7f06f56df"
server_key_version: "encryptionkey@0"
I20260927 22:51:10.713650 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:10.733771 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:10.737772 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:10.737984 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:10.754801 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39657
I20260927 22:51:10.754846 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39657 every 8 connection(s)
I20260927 22:51:10.756085 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:10.759491 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:10.763164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 930
I20260927 22:51:10.763419 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:10.764068 388 external_mini_cluster.cc:1428] Setting key 5ff5b1e6433de44b526f853ad4099a8b
I20260927 22:51:10.771224 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313: Bootstrap starting.
I20260927 22:51:10.774634 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:10.775822 997 log.cc:824] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:10.778591 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313: No bootstrap required, opened a new log
Sep 27 22:51:10 dist-test-slave-df53 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:10.788372 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:10.765995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.0.97.62:39657)
0927 22:51:10.766807 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:10.766823 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:10.766874 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:51:10.769564 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:10.769615 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:10.770084 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:10.770521 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:10.771763 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:10.773753 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:10.775006 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:10.776018 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:10.776202 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:10.780050 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:10.780060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:10.780082 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:10.780128 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:10.783944 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:10.784710 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:10.784719 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:10.784732 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:10.784941 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:10.785257 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:10.785261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:10.785265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:10.785930 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:10.786239 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:10.786627 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 22:51:10.787076 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":337,"threads_started":1}
I20260927 22:51:10.789212 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60ab767f0954afd88109b1f7c407313" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260927 22:51:10.789543 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:10.789655 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60ab767f0954afd88109b1f7c407313, State: Initialized, Role: FOLLOWER
I20260927 22:51:10.790241 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60ab767f0954afd88109b1f7c407313" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260927 22:51:10.790488 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:10.790644 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:10.790825 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:10.792991 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60ab767f0954afd88109b1f7c407313" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260927 22:51:10.793457 997 leader_election.cc:302] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a60ab767f0954afd88109b1f7c407313; no voters:
I20260927 22:51:10.794093 997 leader_election.cc:288] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:10.794221 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:10.795630 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [term 1 LEADER]: Becoming Leader. State: Replica: a60ab767f0954afd88109b1f7c407313, State: Running, Role: LEADER
I20260927 22:51:10.796162 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:10.796177 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60ab767f0954afd88109b1f7c407313" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260927 22:51:10.800117 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a60ab767f0954afd88109b1f7c407313. Latest consensus state: current_term: 1 leader_uuid: "a60ab767f0954afd88109b1f7c407313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60ab767f0954afd88109b1f7c407313" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } } }
I20260927 22:51:10.800498 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a60ab767f0954afd88109b1f7c407313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60ab767f0954afd88109b1f7c407313" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } } }
I20260927 22:51:10.800799 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:10.800498 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:10.801744 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:10.807565 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:10.816579 1011 catalog_manager.cc:1347] Generated new cluster ID: 3908061974414f67a8786fcdcaf6b351
I20260927 22:51:10.816715 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:10.828395 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:10.830165 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:10.836836 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a60ab767f0954afd88109b1f7c407313: Generated new TSK 0
I20260927 22:51:10.837760 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:10.882129 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:11.196058 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:11.196436 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:11.196609 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:11.204002 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:11.204138 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:11.204322 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:11.214126 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:11.215314 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:11.216599 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:11.229372 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:11.229424 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:11.229456 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:11.230232 1023 server_base.cc:1034] running on GCE node
I20260927 22:51:11.230820 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:11.231647 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:11.232867 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790549471232817 us; error 63 us; skew 500 ppm
I20260927 22:51:11.233173 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:11 dist-test-slave-df53 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:11.236127 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:11.237460 1023 webserver.cc:460] Webserver started at http://127.0.97.1:37547/ using document root <none> and password file <none>
I20260927 22:51:11.238125 1023 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:11.238278 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:11.238566 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:11.241281 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "1ba171cc279a4c3b9225ff35535e9c80"
format_stamp: "Formatted at 2026-09-27 22:51:11 on dist-test-slave-df53"
server_key: "781685ed5893d3623907753d687d0501"
server_key_iv: "3bb71f64a3ea4a3ea0b7dfbb79bb2e55"
server_key_version: "encryptionkey@0"
I20260927 22:51:11.242061 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1ba171cc279a4c3b9225ff35535e9c80"
format_stamp: "Formatted at 2026-09-27 22:51:11 on dist-test-slave-df53"
server_key: "781685ed5893d3623907753d687d0501"
server_key_iv: "3bb71f64a3ea4a3ea0b7dfbb79bb2e55"
server_key_version: "encryptionkey@0"
I20260927 22:51:11.246810 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 22:51:11.249984 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:11.251279 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:51:11.251401 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "1ba171cc279a4c3b9225ff35535e9c80"
format_stamp: "Formatted at 2026-09-27 22:51:11 on dist-test-slave-df53"
server_key: "781685ed5893d3623907753d687d0501"
server_key_iv: "3bb71f64a3ea4a3ea0b7dfbb79bb2e55"
server_key_version: "encryptionkey@0"
I20260927 22:51:11.251544 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:11.263700 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:11.267737 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:11.268064 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:11.269268 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:11.270957 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:11.271106 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 22:51:11.271252 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:11.271312 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:11.301571 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37407
I20260927 22:51:11.301657 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37407 every 8 connection(s)
I20260927 22:51:11.302861 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:11.308243 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 1023
I20260927 22:51:11.308462 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:11.309172 388 external_mini_cluster.cc:1428] Setting key 523cafc772b9f948132d5f1742572f2b
Sep 27 22:51:11 dist-test-slave-df53 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:11.323007 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:11.305171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51635 (local address 127.0.97.62:39657)
0927 22:51:11.305655 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:11.305660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:11.306468 (+ 808us) server_negotiation.cc:407] Connection header received
0927 22:51:11.308382 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:11.308388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:11.308477 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:11.308608 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:11.310185 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:11.310884 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:11.312551 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:11.312970 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:11.313110 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:11.316387 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:11.316392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:11.316395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:11.316425 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:11.319792 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:11.320484 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:11.320492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:11.320499 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:11.320643 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:11.321099 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:11.321106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:11.321119 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:11.321507 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:11.321691 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:11.322345 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 22:51:11.322597 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":236,"threads_started":1}
I20260927 22:51:11.323606 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:11.305576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:51635)
0927 22:51:11.306315 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:11.306351 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:11.308051 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:11.308843 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:11.308851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:11.309408 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:11.310017 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:11.310034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:11.311029 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:11.311041 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:11.312411 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:11.312423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:11.313187 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:11.313195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:11.313429 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:11.314150 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:11.314170 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:11.316192 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:11.319938 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:11.319945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:11.319956 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:11.320349 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:11.320780 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:11.320786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:11.320791 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:11.320974 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:11.321679 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:11.321685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:11.321976 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 22:51:11.322189 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 22:51:11.322529 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":256,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:51:11.325079 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39657
I20260927 22:51:11.325399 1151 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:11.326073 1151 heartbeater.cc:511] Master 127.0.97.62:39657 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:51:11.328168 962 ts_manager.cc:194] Registered new tserver with Master: 1ba171cc279a4c3b9225ff35535e9c80 (127.0.97.1:37407)
I20260927 22:51:11.330372 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:11.348560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:11.653126 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:11.653431 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:11.653626 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:11.660658 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:11.660796 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:11.660977 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:11.670156 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:11.671342 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:11.672598 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:11.684902 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:11.684935 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:11.684963 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:11.686411 1157 server_base.cc:1034] running on GCE node
I20260927 22:51:11.686990 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:11.687801 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:11.689019 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790549471688983 us; error 53 us; skew 500 ppm
I20260927 22:51:11.689324 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:11 dist-test-slave-df53 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:11.692466 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:11.693806 1157 webserver.cc:460] Webserver started at http://127.0.97.2:39285/ using document root <none> and password file <none>
I20260927 22:51:11.694427 1157 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:11.694566 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:11.694865 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:11.697670 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02"
format_stamp: "Formatted at 2026-09-27 22:51:11 on dist-test-slave-df53"
server_key: "0c8ff189172dc130f2b521da84393cb8"
server_key_iv: "2efe3f80829de18d325ad04d9fa2ed3d"
server_key_version: "encryptionkey@0"
I20260927 22:51:11.698417 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02"
format_stamp: "Formatted at 2026-09-27 22:51:11 on dist-test-slave-df53"
server_key: "0c8ff189172dc130f2b521da84393cb8"
server_key_iv: "2efe3f80829de18d325ad04d9fa2ed3d"
server_key_version: "encryptionkey@0"
I20260927 22:51:11.703315 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:51:11.706492 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:11.707708 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:51:11.707873 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02"
format_stamp: "Formatted at 2026-09-27 22:51:11 on dist-test-slave-df53"
server_key: "0c8ff189172dc130f2b521da84393cb8"
server_key_iv: "2efe3f80829de18d325ad04d9fa2ed3d"
server_key_version: "encryptionkey@0"
I20260927 22:51:11.708055 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:11.731226 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:11.735385 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:11.735661 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:11.736547 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:11.737805 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:11.737907 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:11.738049 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:11.738121 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:11.771638 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33243
I20260927 22:51:11.771673 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33243 every 8 connection(s)
I20260927 22:51:11.772868 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:11.782536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 1157
I20260927 22:51:11.782734 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:11.783133 388 external_mini_cluster.cc:1428] Setting key 26a5dba33d07eb1ad89f0bf0ae131692
Sep 27 22:51:11 dist-test-slave-df53 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:11.791640 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:11.775322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43287 (local address 127.0.97.62:39657)
0927 22:51:11.775513 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:11.775519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:11.776680 (+ 1161us) server_negotiation.cc:407] Connection header received
0927 22:51:11.778462 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:11.778467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:11.778531 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:11.778619 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:11.779996 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:11.780709 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:11.782232 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:11.782601 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:11.782734 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:11.786292 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:11.786296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:11.786300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:11.786328 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:11.788752 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:11.789427 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:11.789434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:11.789442 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:11.789575 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:11.790033 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:11.790038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:11.790042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:11.790301 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:11.790459 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:11.791106 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 22:51:11.791324 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:51:11.792557 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:11.775850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:43287)
0927 22:51:11.776519 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:11.776564 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:11.778221 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:11.778822 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:11.778830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:11.779319 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:11.779856 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:11.779872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:11.780829 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:11.780832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:11.782098 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:11.782109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:11.782858 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:11.782867 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:11.783070 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:11.783774 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:11.783795 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:11.786077 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:11.788886 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:11.788895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:11.788911 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:11.789305 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:11.789737 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:11.789742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:11.789745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:11.789916 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:11.790434 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:11.790441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:11.790761 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 22:51:11.791009 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 22:51:11.791494 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":259,"threads_started":1}
I20260927 22:51:11.794005 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39657
I20260927 22:51:11.794341 1285 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:11.795013 1285 heartbeater.cc:511] Master 127.0.97.62:39657 requested a full tablet report, sending...
I20260927 22:51:11.796407 962 ts_manager.cc:194] Registered new tserver with Master: b0f5f7601b5e4c4cb28f3f6f04c01d02 (127.0.97.2:33243)
I20260927 22:51:11.797667 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43287
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:11.820423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:12.125917 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:12.126216 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:12.126379 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:12.137985 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:12.138128 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:12.138343 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:12.156023 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:12.157716 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:12.159428 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:12.174829 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:12.174888 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:12.174790 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:12.175621 1290 server_base.cc:1034] running on GCE node
I20260927 22:51:12.176383 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:12.177645 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:12.178913 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790549472178859 us; error 32 us; skew 500 ppm
I20260927 22:51:12.179325 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:12 dist-test-slave-df53 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:12.182683 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:12.184077 1290 webserver.cc:460] Webserver started at http://127.0.97.3:45897/ using document root <none> and password file <none>
I20260927 22:51:12.184720 1290 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:12.184818 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:12.185127 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:12.187999 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "66db46dc8e684d3f9a94a77d5a30b10b"
format_stamp: "Formatted at 2026-09-27 22:51:12 on dist-test-slave-df53"
server_key: "fd6973b541181d366c302509886fa3b8"
server_key_iv: "2f998514a6e0122252942448214b63c3"
server_key_version: "encryptionkey@0"
I20260927 22:51:12.188776 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "66db46dc8e684d3f9a94a77d5a30b10b"
format_stamp: "Formatted at 2026-09-27 22:51:12 on dist-test-slave-df53"
server_key: "fd6973b541181d366c302509886fa3b8"
server_key_iv: "2f998514a6e0122252942448214b63c3"
server_key_version: "encryptionkey@0"
I20260927 22:51:12.193759 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:51:12.197118 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:12.198423 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:12.198593 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "66db46dc8e684d3f9a94a77d5a30b10b"
format_stamp: "Formatted at 2026-09-27 22:51:12 on dist-test-slave-df53"
server_key: "fd6973b541181d366c302509886fa3b8"
server_key_iv: "2f998514a6e0122252942448214b63c3"
server_key_version: "encryptionkey@0"
I20260927 22:51:12.198793 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:12.221153 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:12.224820 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:12.225092 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:12.226099 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:12.227298 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:12.227401 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:12.227572 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:12.227645 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:12.259603 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43359
I20260927 22:51:12.259692 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43359 every 8 connection(s)
I20260927 22:51:12.260852 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:12.265054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 1290
I20260927 22:51:12.265246 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:12.265810 388 external_mini_cluster.cc:1428] Setting key d743599f6b32371c461a0f23a2458992
Sep 27 22:51:12 dist-test-slave-df53 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:12.281488 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.263408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33781 (local address 127.0.97.62:39657)
0927 22:51:12.263587 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.263592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.264653 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 22:51:12.267174 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.267179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.267273 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.267373 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:12.269036 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.269812 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.271303 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.271952 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.272107 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.275930 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:12.275935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:12.275938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:12.275968 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:12.278427 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.279122 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.279126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.279128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.279235 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.279637 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.279643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.279648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.280021 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:12.280151 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.280912 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.281156 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:51:12.282372 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.263780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:33781)
0927 22:51:12.264478 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:12.264524 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:12.266851 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:12.267585 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:12.267593 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:12.268181 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:12.268853 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.268894 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.269927 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.269931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.271110 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.271121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.272215 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.272223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.272441 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.273128 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:12.273156 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:12.275737 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:12.278550 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:12.278556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:12.278568 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:12.278998 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:12.279341 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:12.279346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:12.279349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:12.279543 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:12.280128 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:12.280135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:12.280501 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 22:51:12.280822 (+ 321us) client_negotiation.cc:240] Negotiation successful
0927 22:51:12.281315 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":301,"threads_started":1}
I20260927 22:51:12.283535 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39657
I20260927 22:51:12.283785 1418 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:12.284376 1418 heartbeater.cc:511] Master 127.0.97.62:39657 requested a full tablet report, sending...
I20260927 22:51:12.285626 962 ts_manager.cc:194] Registered new tserver with Master: 66db46dc8e684d3f9a94a77d5a30b10b (127.0.97.3:43359)
I20260927 22:51:12.286379 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33781
I20260927 22:51:12.293150 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:51:12.314222 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.300040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.0.97.62:39657)
0927 22:51:12.300183 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.300187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.300670 (+ 483us) server_negotiation.cc:407] Connection header received
0927 22:51:12.300823 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.300827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.300885 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.300967 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:12.302893 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.303806 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.305587 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.306133 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.306240 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.308480 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:12.308485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:12.308489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:12.308518 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:12.311644 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.312387 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.312395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.312403 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.312523 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.312834 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.312837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.312839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.313129 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:12.313369 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.313823 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.313982 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:51:12.320698 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:12.328190 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:12.333292 1151 heartbeater.cc:503] Master 127.0.97.62:39657 was elected leader, sending a full tablet report...
I20260927 22:51:12.351943 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.341510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.0.97.3:43359)
0927 22:51:12.342914 (+ 1404us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.342927 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.343488 (+ 561us) server_negotiation.cc:407] Connection header received
0927 22:51:12.343697 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.343708 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.343982 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.344192 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:12.345282 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.346982 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.349579 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.350671 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.350862 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.351108 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.351384 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.351559 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1169,"thread_start_us":972,"threads_started":1}
I20260927 22:51:12.352005 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.340926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43359 (local address 127.0.0.1:44272)
0927 22:51:12.343369 (+ 2443us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:12.343387 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:12.343530 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:12.344503 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:12.344512 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:12.344549 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:12.345000 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.345022 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.347116 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.347123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.349382 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.349395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.350909 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.350916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.351114 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.351190 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 22:51:12.351347 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 22:51:12.351549 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2309,"mutex_wait_us":5,"thread_start_us":2186,"threads_started":1}
I20260927 22:51:12.352207 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.340053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33243 (local address 127.0.0.1:44016)
0927 22:51:12.342027 (+ 1974us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:12.342057 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:12.342201 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:12.344699 (+ 2498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:12.344707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:12.344746 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:12.345215 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.345229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.346754 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.346761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.349423 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.349437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.351360 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.351367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.351538 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.351588 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 22:51:12.351653 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 22:51:12.351776 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1797,"mutex_wait_us":2,"thread_start_us":782,"threads_started":1}
I20260927 22:51:12.353130 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.341001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.0.97.2:33243)
0927 22:51:12.344085 (+ 3084us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.344096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.344126 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:51:12.344224 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.344233 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.344472 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.344703 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:12.345403 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.346587 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.349676 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.351127 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.352428 (+ 1301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.352617 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.352686 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.352837 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2910,"thread_start_us":2804,"threads_started":1}
I20260927 22:51:12.353734 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.339881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37407 (local address 127.0.0.1:44496)
0927 22:51:12.341237 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:12.341279 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:12.341445 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:12.346177 (+ 4732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:12.346182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:12.346207 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:12.346521 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.346533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.348424 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.348428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.350961 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.350974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.352986 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.352992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.353173 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.353215 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:51:12.353302 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 22:51:12.353425 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"thread_start_us":2024,"threads_started":1}
I20260927 22:51:12.354435 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.340252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.0.97.1:37407)
0927 22:51:12.345306 (+ 5054us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.345317 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.345344 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:51:12.345439 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.345446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.345722 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.345913 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:12.346721 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.348245 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.351156 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.352562 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.352682 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.352868 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.353929 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.354134 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4890,"thread_start_us":5491,"threads_started":1}
I20260927 22:51:12.356541 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 6dfa555e5b17485988cd8cc62082ac6c (DEFAULT_TABLE table=test-table [id=03a777d3812746d3b611a149503fcbb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:12.356763 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 6dfa555e5b17485988cd8cc62082ac6c (DEFAULT_TABLE table=test-table [id=03a777d3812746d3b611a149503fcbb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:12.357800 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dfa555e5b17485988cd8cc62082ac6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:12.358255 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dfa555e5b17485988cd8cc62082ac6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:12.357834 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 6dfa555e5b17485988cd8cc62082ac6c (DEFAULT_TABLE table=test-table [id=03a777d3812746d3b611a149503fcbb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:12.359187 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dfa555e5b17485988cd8cc62082ac6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:12.370322 1436 tablet_bootstrap.cc:492] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02: Bootstrap starting.
I20260927 22:51:12.373231 1436 tablet_bootstrap.cc:654] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:12.373687 1437 tablet_bootstrap.cc:492] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b: Bootstrap starting.
I20260927 22:51:12.374532 1436 log.cc:824] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:12.377457 1437 tablet_bootstrap.cc:654] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:12.378907 1437 log.cc:824] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:12.380244 1438 tablet_bootstrap.cc:492] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80: Bootstrap starting.
I20260927 22:51:12.383570 1438 tablet_bootstrap.cc:654] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:12.383972 1436 tablet_bootstrap.cc:492] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02: No bootstrap required, opened a new log
I20260927 22:51:12.384282 1436 ts_tablet_manager.cc:1397] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260927 22:51:12.384397 1437 tablet_bootstrap.cc:492] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b: No bootstrap required, opened a new log
I20260927 22:51:12.384727 1437 ts_tablet_manager.cc:1397] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20260927 22:51:12.384841 1438 log.cc:824] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:12.387725 1438 tablet_bootstrap.cc:492] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80: No bootstrap required, opened a new log
I20260927 22:51:12.387975 1438 ts_tablet_manager.cc:1397] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:51:12.391716 1436 raft_consensus.cc:348] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba171cc279a4c3b9225ff35535e9c80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37407 } } peers { permanent_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33243 } } peers { permanent_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43359 } }
I20260927 22:51:12.392042 1436 raft_consensus.cc:374] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:12.392153 1436 raft_consensus.cc:729] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f5f7601b5e4c4cb28f3f6f04c01d02, State: Initialized, Role: FOLLOWER
I20260927 22:51:12.392683 1436 consensus_queue.cc:260] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba171cc279a4c3b9225ff35535e9c80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37407 } } peers { permanent_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33243 } } peers { permanent_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43359 } }
I20260927 22:51:12.393678 1285 heartbeater.cc:503] Master 127.0.97.62:39657 was elected leader, sending a full tablet report...
I20260927 22:51:12.394304 1436 ts_tablet_manager.cc:1428] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260927 22:51:12.396462 1437 raft_consensus.cc:348] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba171cc279a4c3b9225ff35535e9c80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37407 } } peers { permanent_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33243 } } peers { permanent_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43359 } }
I20260927 22:51:12.396890 1437 raft_consensus.cc:374] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:12.396991 1437 raft_consensus.cc:729] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66db46dc8e684d3f9a94a77d5a30b10b, State: Initialized, Role: FOLLOWER
I20260927 22:51:12.397629 1437 consensus_queue.cc:260] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba171cc279a4c3b9225ff35535e9c80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37407 } } peers { permanent_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33243 } } peers { permanent_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43359 } }
I20260927 22:51:12.398702 1418 heartbeater.cc:503] Master 127.0.97.62:39657 was elected leader, sending a full tablet report...
I20260927 22:51:12.399333 1437 ts_tablet_manager.cc:1428] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260927 22:51:12.399662 1438 raft_consensus.cc:348] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba171cc279a4c3b9225ff35535e9c80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37407 } } peers { permanent_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33243 } } peers { permanent_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43359 } }
I20260927 22:51:12.400076 1438 raft_consensus.cc:374] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:12.400197 1438 raft_consensus.cc:729] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba171cc279a4c3b9225ff35535e9c80, State: Initialized, Role: FOLLOWER
I20260927 22:51:12.400832 1438 consensus_queue.cc:260] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba171cc279a4c3b9225ff35535e9c80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37407 } } peers { permanent_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33243 } } peers { permanent_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43359 } }
I20260927 22:51:12.402576 1438 ts_tablet_manager.cc:1428] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260927 22:51:12.496251 1442 raft_consensus.cc:482] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:12.496621 1442 raft_consensus.cc:504] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba171cc279a4c3b9225ff35535e9c80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37407 } } peers { permanent_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33243 } } peers { permanent_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43359 } }
I20260927 22:51:12.498870 1442 leader_election.cc:288] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ba171cc279a4c3b9225ff35535e9c80 (127.0.97.1:37407), 66db46dc8e684d3f9a94a77d5a30b10b (127.0.97.3:43359)
I20260927 22:51:12.505945 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.499335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39703 (local address 127.0.97.3:43359)
0927 22:51:12.499535 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.499543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.500314 (+ 771us) server_negotiation.cc:407] Connection header received
0927 22:51:12.500477 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.500481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.500566 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.500680 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:12.501325 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.502315 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.504216 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.504983 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.505078 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.505208 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.505547 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.505696 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:51:12.505975 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.499189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39533 (local address 127.0.97.1:37407)
0927 22:51:12.499403 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.499410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.500201 (+ 791us) server_negotiation.cc:407] Connection header received
0927 22:51:12.500378 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.500385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.500466 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.500540 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:12.501214 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.502136 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.504271 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.505007 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.505163 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.505260 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.505630 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.505744 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:51:12.506098 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.499218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37407 (local address 127.0.97.2:39533)
0927 22:51:12.500069 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:12.500084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:12.500224 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:12.500578 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:12.500582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:12.500613 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:12.501050 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.501059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.502315 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.502322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.504094 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.504104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.505269 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.505274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.505390 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.505435 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:51:12.505543 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:51:12.505684 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":62,"thread_start_us":458,"threads_started":1}
I20260927 22:51:12.506807 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.499274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43359 (local address 127.0.97.2:39703)
0927 22:51:12.500185 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:12.500203 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:12.500354 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:12.500717 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:12.500720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:12.500747 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:12.501173 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.501184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.502432 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.502436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.504058 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:12.504075 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.505210 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.505217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:12.505379 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.505419 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:51:12.505489 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 22:51:12.505652 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":8,"thread_start_us":552,"threads_started":1}
I20260927 22:51:12.507154 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfa555e5b17485988cd8cc62082ac6c" candidate_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba171cc279a4c3b9225ff35535e9c80" is_pre_election: true
I20260927 22:51:12.507768 1106 raft_consensus.cc:2454] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f5f7601b5e4c4cb28f3f6f04c01d02 in term 0.
I20260927 22:51:12.507702 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfa555e5b17485988cd8cc62082ac6c" candidate_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" is_pre_election: true
I20260927 22:51:12.508212 1373 raft_consensus.cc:2454] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f5f7601b5e4c4cb28f3f6f04c01d02 in term 0.
I20260927 22:51:12.508467 1174 leader_election.cc:302] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ba171cc279a4c3b9225ff35535e9c80, b0f5f7601b5e4c4cb28f3f6f04c01d02; no voters:
I20260927 22:51:12.508852 1442 raft_consensus.cc:2781] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:12.508967 1442 raft_consensus.cc:482] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:12.509080 1442 raft_consensus.cc:3037] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:12.511255 1442 raft_consensus.cc:504] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba171cc279a4c3b9225ff35535e9c80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37407 } } peers { permanent_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33243 } } peers { permanent_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43359 } }
I20260927 22:51:12.511878 1442 leader_election.cc:288] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [CANDIDATE]: Term 1 election: Requested vote from peers 1ba171cc279a4c3b9225ff35535e9c80 (127.0.97.1:37407), 66db46dc8e684d3f9a94a77d5a30b10b (127.0.97.3:43359)
I20260927 22:51:12.512331 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfa555e5b17485988cd8cc62082ac6c" candidate_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba171cc279a4c3b9225ff35535e9c80"
W20260927 22:51:12.512445 1419 tablet.cc:2369] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:12.512396 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfa555e5b17485988cd8cc62082ac6c" candidate_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66db46dc8e684d3f9a94a77d5a30b10b"
I20260927 22:51:12.512512 1106 raft_consensus.cc:3037] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:12.512609 1373 raft_consensus.cc:3037] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:12.514885 1106 raft_consensus.cc:2454] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f5f7601b5e4c4cb28f3f6f04c01d02 in term 1.
I20260927 22:51:12.515131 1373 raft_consensus.cc:2454] T 6dfa555e5b17485988cd8cc62082ac6c P 66db46dc8e684d3f9a94a77d5a30b10b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f5f7601b5e4c4cb28f3f6f04c01d02 in term 1.
I20260927 22:51:12.515268 1174 leader_election.cc:302] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ba171cc279a4c3b9225ff35535e9c80, b0f5f7601b5e4c4cb28f3f6f04c01d02; no voters:
I20260927 22:51:12.515549 1442 raft_consensus.cc:2781] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:12.515974 1442 raft_consensus.cc:686] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [term 1 LEADER]: Becoming Leader. State: Replica: b0f5f7601b5e4c4cb28f3f6f04c01d02, State: Running, Role: LEADER
I20260927 22:51:12.516422 1442 consensus_queue.cc:237] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba171cc279a4c3b9225ff35535e9c80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37407 } } peers { permanent_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33243 } } peers { permanent_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43359 } }
I20260927 22:51:12.521008 961 catalog_manager.cc:5495] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0f5f7601b5e4c4cb28f3f6f04c01d02 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba171cc279a4c3b9225ff35535e9c80" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0f5f7601b5e4c4cb28f3f6f04c01d02" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66db46dc8e684d3f9a94a77d5a30b10b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43359 } health_report { overall_health: UNKNOWN } } }
W20260927 22:51:12.525434 1286 tablet.cc:2369] T 6dfa555e5b17485988cd8cc62082ac6c P b0f5f7601b5e4c4cb28f3f6f04c01d02: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 22:51:12 dist-test-slave-df53 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:12.555051 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.536462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.0.97.1:37407)
0927 22:51:12.536634 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.536638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.537179 (+ 541us) server_negotiation.cc:407] Connection header received
0927 22:51:12.537315 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.537319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.537399 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.537481 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:12.538930 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.539663 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.540851 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.541306 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.541404 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.548899 (+ 7495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:12.548921 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:12.548948 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:12.548982 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:12.551995 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.552587 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.552594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.552596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.552741 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.553102 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.553105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.553107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.553948 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:12.554180 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.554448 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.554634 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:51:12.555022 1152 tablet.cc:2369] T 6dfa555e5b17485988cd8cc62082ac6c P 1ba171cc279a4c3b9225ff35535e9c80: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 22:51:12 dist-test-slave-df53 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549472, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:51:12 dist-test-slave-df53 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549472, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:12.598919 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.585374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.0.97.1:37407)
0927 22:51:12.585618 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.585626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.585895 (+ 269us) server_negotiation.cc:407] Connection header received
0927 22:51:12.586117 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.586122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.586188 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.586296 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:12.587561 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.588285 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.589538 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.590013 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.590141 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.593733 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:12.593738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:12.593741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:12.593773 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:12.596353 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.597028 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.597036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.597039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.597183 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.597549 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.597556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.597559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.597862 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:12.598134 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.598380 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.598589 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 22:51:12 dist-test-slave-df53 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:51:12 dist-test-slave-df53 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:12.641749 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.628591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.0.97.1:37407)
0927 22:51:12.628853 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.628860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.629124 (+ 264us) server_negotiation.cc:407] Connection header received
0927 22:51:12.629338 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.629343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.629403 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.629506 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:12.630795 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.631449 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.632625 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.633044 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.633170 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.636879 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:12.636884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:12.636887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:12.636915 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:12.639208 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.639788 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.639795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.639799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.639914 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.640323 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.640328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.640331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.640604 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:12.640869 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.641130 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.641357 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:51:12.643359 1066 tablet_service.cc:2908] Rejecting scan request for tablet 6dfa555e5b17485988cd8cc62082ac6c: Uninitialized: clean time has not yet been initialized
I20260927 22:51:12.646409 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 1023
I20260927 22:51:12.658241 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.647301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.0.97.1:37407)
0927 22:51:12.647452 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.647455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.647798 (+ 343us) server_negotiation.cc:407] Connection header received
0927 22:51:12.647893 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.647897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.647959 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.648036 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:12.649080 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.649790 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.650917 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.651285 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.651382 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.653712 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:12.653717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:12.653720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:12.653746 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:12.656263 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.656795 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.656798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.656800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.656900 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.657187 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.657190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.657191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.657431 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:12.657710 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.657872 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.658014 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:51:12.658866 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:12.746886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 1023
I20260927 22:51:12.763972 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 1157
Sep 27 22:51:12 dist-test-slave-df53 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:12.779945 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.764748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.0.97.2:33243)
0927 22:51:12.764912 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.764916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.765081 (+ 165us) server_negotiation.cc:407] Connection header received
0927 22:51:12.765338 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.765343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.765425 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.765536 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:12.766710 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.767407 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.768685 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.769092 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.769223 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.773555 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:12.773566 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:12.773582 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:12.773645 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:12.777284 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.777954 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.777963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.777967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.778150 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.778475 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.778480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.778484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.778882 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:12.779159 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.779364 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.779604 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 22:51:12.780747 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:12.869033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 1157
I20260927 22:51:12.885571 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 1290
Sep 27 22:51:12 dist-test-slave-df53 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:51:12.899564 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:12.886331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.0.97.3:43359)
0927 22:51:12.886493 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:12.886497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:12.886557 (+ 60us) server_negotiation.cc:407] Connection header received
0927 22:51:12.886711 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:12.886715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:12.886782 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:12.886867 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:12.888030 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.888804 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.890052 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:12.890414 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:12.890509 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:12.894415 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:12.894427 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:12.894442 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:12.894473 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:12.897089 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.897797 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.897804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.897806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.897919 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:12.898234 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:12.898237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:12.898239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:12.898546 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:12.898761 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:12.899062 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 22:51:12.899216 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:51:12.900269 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:12.985688 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 1290
I20260927 22:51:13.001771 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 930
I20260927 22:51:13.002889 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:13.107373 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 930
2026-09-27T22:51:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3997 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:13 dist-test-slave-df53 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:13 dist-test-slave-df53 krb5kdc[1466](info): set up 2 sockets
Sep 27 22:51:13 dist-test-slave-df53 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260927 22:51:14.193781 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:14 dist-test-slave-df53 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:14.364856 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38563
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:14.670323 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:14.670619 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:14.670782 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:14.677763 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:14.677901 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:51:14.677989 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:14.678107 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:14.678187 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:14.678277 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:14.687788 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38563
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:14.689023 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:14.690378 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:14.701711 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:14.701743 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:14.702023 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:14.702394 1479 server_base.cc:1034] running on GCE node
I20260927 22:51:14.703110 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:14.704385 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:14.705643 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790549474705537 us; error 35 us; skew 500 ppm
I20260927 22:51:14.705992 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:14 dist-test-slave-df53 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:14.708847 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:14.710176 1479 webserver.cc:460] Webserver started at http://127.0.97.62:33025/ using document root <none> and password file <none>
I20260927 22:51:14.710769 1479 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:14.710850 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:14.711094 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:14.713820 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "5265b9ff0c084b8ea87abd1f5331921b"
format_stamp: "Formatted at 2026-09-27 22:51:14 on dist-test-slave-df53"
server_key: "2b951179733b06fb8a270c4f5780db11"
server_key_iv: "6fceac2402252407166795bbae946678"
server_key_version: "encryptionkey@0"
I20260927 22:51:14.714564 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "5265b9ff0c084b8ea87abd1f5331921b"
format_stamp: "Formatted at 2026-09-27 22:51:14 on dist-test-slave-df53"
server_key: "2b951179733b06fb8a270c4f5780db11"
server_key_iv: "6fceac2402252407166795bbae946678"
server_key_version: "encryptionkey@0"
I20260927 22:51:14.719286 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:51:14.722623 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:14.723848 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:51:14.724030 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "5265b9ff0c084b8ea87abd1f5331921b"
format_stamp: "Formatted at 2026-09-27 22:51:14 on dist-test-slave-df53"
server_key: "2b951179733b06fb8a270c4f5780db11"
server_key_iv: "6fceac2402252407166795bbae946678"
server_key_version: "encryptionkey@0"
I20260927 22:51:14.724228 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:14.762938 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:14.767285 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:14.767553 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:14.788993 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38563
I20260927 22:51:14.789000 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38563 every 8 connection(s)
I20260927 22:51:14.790433 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:14.794179 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:14.799675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 1479
I20260927 22:51:14.799901 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:14.800509 388 external_mini_cluster.cc:1428] Setting key 01bf3b5359112cd1a00d26657daaf13b
I20260927 22:51:14.804322 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b: Bootstrap starting.
I20260927 22:51:14.807876 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:14.808892 1556 log.cc:824] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:14.811693 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b: No bootstrap required, opened a new log
Sep 27 22:51:14 dist-test-slave-df53 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:14.820281 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265b9ff0c084b8ea87abd1f5331921b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38563 } }
I20260927 22:51:14.820670 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:14.820777 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5265b9ff0c084b8ea87abd1f5331921b, State: Initialized, Role: FOLLOWER
I20260927 22:51:14.821426 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265b9ff0c084b8ea87abd1f5331921b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38563 } }
I20260927 22:51:14.821693 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:14.821862 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:14.822043 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:14.823069 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:14.802387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.0.97.62:38563)
0927 22:51:14.803141 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:14.803152 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:14.803190 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:51:14.804846 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:14.804868 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:14.805236 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:14.805541 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:14.807310 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:14.808638 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:14.810226 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:14.810857 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:14.810990 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:14.815577 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:14.815585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:14.815598 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:14.815632 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:14.818854 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:14.819579 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:14.819594 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:14.819600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:14.819754 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:14.820119 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:14.820124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:14.820129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:14.820678 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:14.821000 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:14.821400 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 22:51:14.822054 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":362,"threads_started":1}
I20260927 22:51:14.824360 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265b9ff0c084b8ea87abd1f5331921b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38563 } }
I20260927 22:51:14.824826 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5265b9ff0c084b8ea87abd1f5331921b; no voters:
I20260927 22:51:14.825469 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:14.825613 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:14.826382 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [term 1 LEADER]: Becoming Leader. State: Replica: 5265b9ff0c084b8ea87abd1f5331921b, State: Running, Role: LEADER
I20260927 22:51:14.826934 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265b9ff0c084b8ea87abd1f5331921b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38563 } }
I20260927 22:51:14.827680 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:14.830060 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5265b9ff0c084b8ea87abd1f5331921b. Latest consensus state: current_term: 1 leader_uuid: "5265b9ff0c084b8ea87abd1f5331921b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265b9ff0c084b8ea87abd1f5331921b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38563 } } }
I20260927 22:51:14.830085 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5265b9ff0c084b8ea87abd1f5331921b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5265b9ff0c084b8ea87abd1f5331921b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38563 } } }
I20260927 22:51:14.830509 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:14.830513 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:14.834995 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:14.840224 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:14.848836 1568 catalog_manager.cc:1347] Generated new cluster ID: d9c205be8e9a4f81a23133470e76ebae
I20260927 22:51:14.848946 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:14.870822 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:14.872284 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:14.879572 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b: Generated new TSK 0
I20260927 22:51:14.880357 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:14.928753 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38563
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:15.239642 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:15.239967 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:15.240139 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:15.247135 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:15.247272 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:15.247423 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:51:15.247560 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:15.256755 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:15.257953 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:15.259236 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:15.272374 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:15.272441 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:15.272459 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:15.278980 1582 server_base.cc:1034] running on GCE node
I20260927 22:51:15.279527 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:15.280362 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:15.281574 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790549475281520 us; error 38 us; skew 500 ppm
I20260927 22:51:15.281920 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:15 dist-test-slave-df53 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:15.285769 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:15.287508 1582 webserver.cc:460] Webserver started at http://127.0.97.1:33845/ using document root <none> and password file <none>
I20260927 22:51:15.288144 1582 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:15.288226 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:15.288492 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:15.291229 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759"
format_stamp: "Formatted at 2026-09-27 22:51:15 on dist-test-slave-df53"
server_key: "6e4a391c9fc1b2c3d17a37fc8e81c033"
server_key_iv: "e4edb893b5cc256f554863ad231e2840"
server_key_version: "encryptionkey@0"
I20260927 22:51:15.292012 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759"
format_stamp: "Formatted at 2026-09-27 22:51:15 on dist-test-slave-df53"
server_key: "6e4a391c9fc1b2c3d17a37fc8e81c033"
server_key_iv: "e4edb893b5cc256f554863ad231e2840"
server_key_version: "encryptionkey@0"
I20260927 22:51:15.296799 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 22:51:15.300083 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:15.301249 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:51:15.301420 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759"
format_stamp: "Formatted at 2026-09-27 22:51:15 on dist-test-slave-df53"
server_key: "6e4a391c9fc1b2c3d17a37fc8e81c033"
server_key_iv: "e4edb893b5cc256f554863ad231e2840"
server_key_version: "encryptionkey@0"
I20260927 22:51:15.301662 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:15.316915 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:15.321066 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:15.321337 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:15.324028 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:15.324124 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:15.324290 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:15.324381 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:51:15 dist-test-slave-df53 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:15.346194 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:15.324529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38377 (local address 127.0.97.62:38563)
0927 22:51:15.325403 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:15.325408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:15.326236 (+ 828us) server_negotiation.cc:407] Connection header received
0927 22:51:15.328512 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:15.328517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:15.328603 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:15.328738 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:15.330949 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.331826 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.333419 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.333890 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.334039 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:15.339247 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:15.339264 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:15.339271 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:15.339320 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:15.342539 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:15.343709 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:15.343715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:15.343718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:15.343890 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:15.344402 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:15.344408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:15.344411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:15.344679 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:15.344864 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:15.345571 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 22:51:15.345835 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":677,"threads_started":1}
I20260927 22:51:15.347142 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:15.325213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38563 (local address 127.0.97.1:38377)
0927 22:51:15.326084 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:15.326120 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:15.328176 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:15.328984 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:15.328992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:15.329839 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:15.330761 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:15.330793 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.331984 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.331989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:15.333272 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:15.333283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.334334 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.334339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:15.334616 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:15.335490 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:15.335510 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:15.339020 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:15.342689 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:15.342705 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:15.342724 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:15.343519 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:15.344028 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:15.344038 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:15.344044 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:15.344270 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:15.344838 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:15.344846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:15.345161 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 22:51:15.345483 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 22:51:15.346034 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":439,"threads_started":1}
I20260927 22:51:15.373454 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44279
I20260927 22:51:15.373473 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44279 every 8 connection(s)
I20260927 22:51:15.374514 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:15.358319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38563 (local address 127.0.97.1:60041)
0927 22:51:15.359118 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:15.359141 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:15.359313 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:15.359700 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:15.359703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:15.360091 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:15.360384 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:15.360392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.362024 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.362032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:15.364305 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:15.364324 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.365248 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.365254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:15.365441 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:15.366196 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:15.366222 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:15.367673 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:15.371759 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:15.371766 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:15.371769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:15.372317 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:15.372732 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:15.372737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:15.372741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:15.372968 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:15.373674 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:15.373681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:15.373869 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 22:51:15.373993 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 22:51:15.374213 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":455,"threads_started":1}
I20260927 22:51:15.374826 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:15.358273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60041 (local address 127.0.97.62:38563)
0927 22:51:15.358451 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:15.358458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:15.359240 (+ 782us) server_negotiation.cc:407] Connection header received
0927 22:51:15.359447 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:15.359455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:15.359554 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:15.359672 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:15.360529 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.361791 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.364475 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.365056 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.365148 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:15.367895 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:15.367903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:15.367910 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:15.367955 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:15.371636 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:15.372454 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:15.372459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:15.372462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:15.372612 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:15.373101 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:15.373108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:15.373115 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:15.373509 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:15.373683 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:15.374111 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 22:51:15.374331 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:51:15.375089 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:15.376157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 1582
I20260927 22:51:15.376403 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:15.377086 388 external_mini_cluster.cc:1428] Setting key 44601336b5eb98e9fb501dd6a4abea19
I20260927 22:51:15.377130 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38563
I20260927 22:51:15.379618 1718 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:15.380435 1718 heartbeater.cc:511] Master 127.0.97.62:38563 requested a full tablet report, sending...
I20260927 22:51:15.382361 1511 ts_manager.cc:194] Registered new tserver with Master: ffd0304cc23a4d57a31ee0dc4fbc7759 (127.0.97.1:44279)
I20260927 22:51:15.384135 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38377
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:15.416353 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38563
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:15.724753 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:15.724996 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:15.725149 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:15.732246 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:15.732336 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:15.732503 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:51:15.732618 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:15.741897 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:15.743024 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:15.744246 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:15.756858 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:15.756902 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:15.756927 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:15.758029 1722 server_base.cc:1034] running on GCE node
I20260927 22:51:15.758695 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:15.759690 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:15.760932 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790549475760862 us; error 36 us; skew 500 ppm
I20260927 22:51:15.761184 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:15 dist-test-slave-df53 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:15.764161 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:15.765514 1722 webserver.cc:460] Webserver started at http://127.0.97.2:42219/ using document root <none> and password file <none>
I20260927 22:51:15.766250 1722 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:15.766358 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:15.766680 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:15.769371 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "4a670118694b4833a989e1e663a03a4c"
format_stamp: "Formatted at 2026-09-27 22:51:15 on dist-test-slave-df53"
server_key: "cac7363a6fca6560968939b1157724a5"
server_key_iv: "04d60149dd78bb5aa75e9f63dfe3891e"
server_key_version: "encryptionkey@0"
I20260927 22:51:15.770174 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4a670118694b4833a989e1e663a03a4c"
format_stamp: "Formatted at 2026-09-27 22:51:15 on dist-test-slave-df53"
server_key: "cac7363a6fca6560968939b1157724a5"
server_key_iv: "04d60149dd78bb5aa75e9f63dfe3891e"
server_key_version: "encryptionkey@0"
I20260927 22:51:15.775031 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:51:15.778203 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:15.779386 1722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:51:15.779546 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "4a670118694b4833a989e1e663a03a4c"
format_stamp: "Formatted at 2026-09-27 22:51:15 on dist-test-slave-df53"
server_key: "cac7363a6fca6560968939b1157724a5"
server_key_iv: "04d60149dd78bb5aa75e9f63dfe3891e"
server_key_version: "encryptionkey@0"
I20260927 22:51:15.779747 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:15.792670 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:15.796643 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:15.796926 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:15.799628 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:15.799747 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:15.799926 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:15.800022 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:51:15 dist-test-slave-df53 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:15.825027 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:15.800818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54589 (local address 127.0.97.62:38563)
0927 22:51:15.801000 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:15.801007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:15.802489 (+ 1482us) server_negotiation.cc:407] Connection header received
0927 22:51:15.805391 (+ 2902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:15.805397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:15.805479 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:15.805556 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:15.808080 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.809435 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.812788 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.813556 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.813727 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:15.818740 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:15.818745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:15.818748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:15.818780 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:15.822135 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:15.822847 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:15.822857 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:15.822859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:15.822968 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:15.823398 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:15.823403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:15.823406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:15.823662 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:15.823817 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:15.824476 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 22:51:15.824680 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:51:15.826404 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:15.801348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38563 (local address 127.0.97.2:54589)
0927 22:51:15.802339 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:15.802386 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:15.805043 (+ 2657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:15.806143 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:15.806160 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:15.806993 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:15.807867 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:15.807901 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.809642 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.809650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:15.812541 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:15.812579 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.813854 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.813866 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:15.814090 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:15.814979 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:15.815018 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:15.818429 (+ 3411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:15.822275 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:15.822283 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:15.822299 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:15.822713 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:15.823081 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:15.823086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:15.823090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:15.823283 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:15.823810 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:15.823818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:15.824130 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 22:51:15.824382 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:51:15.824891 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":419,"threads_started":1}
I20260927 22:51:15.845458 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:15.832047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55551 (local address 127.0.97.62:38563)
0927 22:51:15.832252 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:15.832261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:15.832791 (+ 530us) server_negotiation.cc:407] Connection header received
0927 22:51:15.832929 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:15.832934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:15.833011 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:15.833128 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:15.834556 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.835490 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.837044 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.837438 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.837570 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:15.839363 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:15.839368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:15.839371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:15.839397 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:15.842479 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:15.843523 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:15.843530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:15.843536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:15.843680 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:15.844069 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:15.844076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:15.844082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:15.844379 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:15.844529 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:15.844933 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 22:51:15.845130 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:51:15.845669 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:15.832046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38563 (local address 127.0.97.2:55551)
0927 22:51:15.832694 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:15.832716 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:15.832823 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:15.833186 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:15.833193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:15.833878 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:15.834359 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:15.834382 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.835675 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.835678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:15.836896 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:15.836907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:15.837633 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:15.837640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:15.837814 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:15.838462 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:15.838479 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:15.839163 (+ 684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:15.842651 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:15.842656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:15.842661 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:15.843388 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:15.843830 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:15.843833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:15.843836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:15.843952 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:15.844504 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:15.844509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:15.844689 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 22:51:15.844850 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 22:51:15.845165 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":350,"threads_started":1}
I20260927 22:51:15.856400 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37935
I20260927 22:51:15.856418 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37935 every 8 connection(s)
I20260927 22:51:15.857546 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:15.859584 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38563
I20260927 22:51:15.859754 1857 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:15.860333 1857 heartbeater.cc:511] Master 127.0.97.62:38563 requested a full tablet report, sending...
I20260927 22:51:15.861410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 1722
I20260927 22:51:15.861627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:15.861625 1511 ts_manager.cc:194] Registered new tserver with Master: 4a670118694b4833a989e1e663a03a4c (127.0.97.2:37935)
I20260927 22:51:15.862139 388 external_mini_cluster.cc:1428] Setting key e0ed1c1045e04f4abca3139b3f5d0e8f
I20260927 22:51:15.862685 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54589
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:15.900787 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38563
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:16.214907 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:16.215169 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:16.215359 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:16.222385 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:16.222501 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:16.222668 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:51:16.222801 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:16.232461 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:16.233646 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:16.234853 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:16.247402 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:16.247402 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:16.247435 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:16.248523 1861 server_base.cc:1034] running on GCE node
I20260927 22:51:16.249209 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:16.250137 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:16.251379 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790549476251308 us; error 52 us; skew 500 ppm
I20260927 22:51:16.251663 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:16 dist-test-slave-df53 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:16.254570 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:16.255973 1861 webserver.cc:460] Webserver started at http://127.0.97.3:36113/ using document root <none> and password file <none>
I20260927 22:51:16.256608 1861 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:16.256757 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:16.257057 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:16.259774 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "450c3c4c3ab94c11b044cee4780174e9"
format_stamp: "Formatted at 2026-09-27 22:51:16 on dist-test-slave-df53"
server_key: "268d31b9ed0af730e552f5257c2ba235"
server_key_iv: "4cb791231f727967b21d5f00652720ab"
server_key_version: "encryptionkey@0"
I20260927 22:51:16.260509 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "450c3c4c3ab94c11b044cee4780174e9"
format_stamp: "Formatted at 2026-09-27 22:51:16 on dist-test-slave-df53"
server_key: "268d31b9ed0af730e552f5257c2ba235"
server_key_iv: "4cb791231f727967b21d5f00652720ab"
server_key_version: "encryptionkey@0"
I20260927 22:51:16.265149 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260927 22:51:16.268447 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:16.269771 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:16.269945 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "450c3c4c3ab94c11b044cee4780174e9"
format_stamp: "Formatted at 2026-09-27 22:51:16 on dist-test-slave-df53"
server_key: "268d31b9ed0af730e552f5257c2ba235"
server_key_iv: "4cb791231f727967b21d5f00652720ab"
server_key_version: "encryptionkey@0"
I20260927 22:51:16.270153 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:16.311144 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:16.315146 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:16.315433 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:16.317994 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:16.318104 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:16.318224 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:16.318301 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:51:16 dist-test-slave-df53 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:16.343341 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.319359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38563 (local address 127.0.97.3:44869)
0927 22:51:16.320219 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:16.320260 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:16.322693 (+ 2433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:16.323845 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:16.323858 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:16.324589 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:16.325716 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.325746 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.327181 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.327186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.329611 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.329632 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.330713 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.330721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.330952 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.331770 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:16.331807 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:16.335048 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:16.338388 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:16.338396 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:16.338410 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:16.339050 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:16.339483 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:16.339487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:16.339490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:16.339710 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:16.340372 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:16.340381 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:16.340876 (+ 495us) client_negotiation.cc:769] Sending connection context
0927 22:51:16.341234 (+ 358us) client_negotiation.cc:240] Negotiation successful
0927 22:51:16.341796 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":399,"threads_started":1}
I20260927 22:51:16.344173 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.318685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44869 (local address 127.0.97.62:38563)
0927 22:51:16.318888 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:16.318894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:16.320472 (+ 1578us) server_negotiation.cc:407] Connection header received
0927 22:51:16.323041 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:16.323049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:16.323153 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:16.323232 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:16.325915 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.327024 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.329847 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.330442 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.330586 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.335295 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:16.335300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:16.335303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:16.335340 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:16.338246 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:16.339156 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:16.339162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:16.339165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:16.339330 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:16.339845 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:16.339850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:16.339852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:16.340227 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:16.340387 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:16.343786 (+ 3399us) server_negotiation.cc:299] Negotiation successful
0927 22:51:16.343949 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:51:16.370105 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42479
I20260927 22:51:16.370136 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42479 every 8 connection(s)
I20260927 22:51:16.371131 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.355955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40183 (local address 127.0.97.62:38563)
0927 22:51:16.356182 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:16.356190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:16.356787 (+ 597us) server_negotiation.cc:407] Connection header received
0927 22:51:16.356978 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:16.356981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:16.357052 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:16.357137 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:16.358681 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.359463 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.361929 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.362673 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.362823 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.364482 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:16.364492 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:16.364499 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:16.364542 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:16.368357 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:16.369021 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:16.369026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:16.369030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:16.369224 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:16.369645 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:16.369650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:16.369653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:16.369900 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:16.370062 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:16.370561 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 22:51:16.370777 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:51:16.371223 1967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.355939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38563 (local address 127.0.97.3:40183)
0927 22:51:16.356643 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:16.356670 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:16.356853 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:16.357199 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:16.357207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:16.357949 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:16.358504 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.358518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.359595 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.359606 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.361701 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.361723 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.362912 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.362917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.363027 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.363635 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:16.363652 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:16.364300 (+ 648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:16.368472 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:16.368475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:16.368478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:16.368853 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:16.369373 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:16.369380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:16.369385 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:16.369517 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:16.370021 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:16.370025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:16.370271 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 22:51:16.370438 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 22:51:16.370767 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":361,"threads_started":1}
I20260927 22:51:16.371459 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:16.374782 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38563
I20260927 22:51:16.375036 1996 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:16.375645 1996 heartbeater.cc:511] Master 127.0.97.62:38563 requested a full tablet report, sending...
I20260927 22:51:16.376907 1511 ts_manager.cc:194] Registered new tserver with Master: 450c3c4c3ab94c11b044cee4780174e9 (127.0.97.3:42479)
I20260927 22:51:16.377959 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44869
I20260927 22:51:16.378465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 1861
I20260927 22:51:16.378662 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:16.379190 388 external_mini_cluster.cc:1428] Setting key 0ca71b93c720dd1acf78df0f5601881f
I20260927 22:51:16.381841 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:51:16.387353 1718 heartbeater.cc:503] Master 127.0.97.62:38563 was elected leader, sending a full tablet report...
Sep 27 22:51:16 dist-test-slave-df53 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:51:16 dist-test-slave-df53 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:16.425880 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.412332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.0.97.62:38563)
0927 22:51:16.412496 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:16.412501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:16.412656 (+ 155us) server_negotiation.cc:407] Connection header received
0927 22:51:16.412988 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:16.412993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:16.413057 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:16.413158 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:16.414678 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.415419 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.416685 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.417146 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.417248 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.420907 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:16.420912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:16.420915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:16.420945 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:16.423626 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:16.424236 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:16.424241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:16.424243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:16.424361 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:16.424674 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:16.424677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:16.424679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:16.424921 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:16.425083 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:16.425334 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 22:51:16.425485 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:51:16.430083 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:16.436867 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:16.461267 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.449240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.0.97.2:37935)
0927 22:51:16.451413 (+ 2173us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:16.451425 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:16.451459 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:51:16.451574 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:16.451586 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:16.451903 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:16.452998 (+ 1095us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:16.453243 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.455152 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.458871 (+ 3719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.460139 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.460286 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.460494 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:16.460829 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:51:16.460982 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1984,"thread_start_us":1854,"threads_started":1}
I20260927 22:51:16.461320 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.448904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37935 (local address 127.0.0.1:52230)
0927 22:51:16.450723 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:16.450745 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:16.450881 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:16.452186 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:16.452196 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:16.452235 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:16.452718 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.452734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.455929 (+ 3195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.455935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.458680 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.458696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.460361 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.460368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.460572 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.460619 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 22:51:16.460778 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:51:16.460973 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1645,"thread_start_us":6527,"threads_started":1}
I20260927 22:51:16.463620 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.448525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.0.97.1:44279)
0927 22:51:16.456320 (+ 7795us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:16.456328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:16.456354 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:51:16.456446 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:16.456453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:16.456673 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:16.456879 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:16.457638 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.458947 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.460945 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.462679 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.462804 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.463005 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:16.463208 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 22:51:16.463343 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7652,"thread_start_us":7506,"threads_started":1}
I20260927 22:51:16.463634 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.448493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44279 (local address 127.0.0.1:56876)
0927 22:51:16.449798 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:16.449854 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:16.450047 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:16.456879 (+ 6832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:16.456883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:16.456907 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:16.457397 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.457414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.459066 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.459072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.460752 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.460764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.462856 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.462864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.463031 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.463060 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:51:16.463157 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:51:16.463279 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":3,"thread_start_us":704,"threads_started":1}
I20260927 22:51:16.464437 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.450108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42479 (local address 127.0.0.1:50742)
0927 22:51:16.455658 (+ 5550us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:16.455675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:16.455784 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:16.457278 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:16.457282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:16.457306 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:16.457731 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.457743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.459921 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.459927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.462096 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.462110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.463869 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.463875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.464028 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.464057 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:51:16.464118 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 22:51:16.464211 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5436,"thread_start_us":787,"threads_started":1}
I20260927 22:51:16.465502 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.450675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.0.97.3:42479)
0927 22:51:16.456712 (+ 6037us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:16.456718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:16.456741 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:51:16.456833 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:16.456839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:16.457065 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:16.457271 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:16.458159 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.459778 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.462273 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.463638 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.464736 (+ 1098us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.464943 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:16.465016 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 22:51:16.465163 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5905,"thread_start_us":5753,"threads_started":1}
I20260927 22:51:16.465335 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad8a1f6dee54cd480c5dd5b25bba713 (DEFAULT_TABLE table=test-table [id=7fe16411c9b7488f89e42fa85937328e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:16.467037 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad8a1f6dee54cd480c5dd5b25bba713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:16.467458 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad8a1f6dee54cd480c5dd5b25bba713 (DEFAULT_TABLE table=test-table [id=7fe16411c9b7488f89e42fa85937328e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:16.468787 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad8a1f6dee54cd480c5dd5b25bba713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:16.468743 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad8a1f6dee54cd480c5dd5b25bba713 (DEFAULT_TABLE table=test-table [id=7fe16411c9b7488f89e42fa85937328e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:16.470029 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad8a1f6dee54cd480c5dd5b25bba713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:16.479158 2015 tablet_bootstrap.cc:492] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: Bootstrap starting.
I20260927 22:51:16.482146 2015 tablet_bootstrap.cc:654] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:16.483358 2015 log.cc:824] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:16.486482 2016 tablet_bootstrap.cc:492] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: Bootstrap starting.
I20260927 22:51:16.489552 2017 tablet_bootstrap.cc:492] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: Bootstrap starting.
I20260927 22:51:16.491099 2016 tablet_bootstrap.cc:654] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:16.492460 2017 tablet_bootstrap.cc:654] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:16.492686 2016 log.cc:824] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:16.493790 2017 log.cc:824] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:16.495826 2015 tablet_bootstrap.cc:492] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: No bootstrap required, opened a new log
I20260927 22:51:16.496091 2015 ts_tablet_manager.cc:1397] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260927 22:51:16.496855 2016 tablet_bootstrap.cc:492] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: No bootstrap required, opened a new log
I20260927 22:51:16.497073 2017 tablet_bootstrap.cc:492] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: No bootstrap required, opened a new log
I20260927 22:51:16.497098 2016 ts_tablet_manager.cc:1397] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260927 22:51:16.497315 2017 ts_tablet_manager.cc:1397] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:51:16.504416 2016 raft_consensus.cc:348] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } }
I20260927 22:51:16.504832 2016 raft_consensus.cc:374] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:16.504971 2016 raft_consensus.cc:729] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd0304cc23a4d57a31ee0dc4fbc7759, State: Initialized, Role: FOLLOWER
I20260927 22:51:16.505774 2016 consensus_queue.cc:260] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } }
I20260927 22:51:16.507573 2016 ts_tablet_manager.cc:1428] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20260927 22:51:16.507364 2015 raft_consensus.cc:348] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } }
I20260927 22:51:16.507824 2015 raft_consensus.cc:374] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:16.507944 2015 raft_consensus.cc:729] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a670118694b4833a989e1e663a03a4c, State: Initialized, Role: FOLLOWER
I20260927 22:51:16.507967 2017 raft_consensus.cc:348] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } }
I20260927 22:51:16.508419 2017 raft_consensus.cc:374] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:16.508553 2017 raft_consensus.cc:729] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450c3c4c3ab94c11b044cee4780174e9, State: Initialized, Role: FOLLOWER
I20260927 22:51:16.508531 2015 consensus_queue.cc:260] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } }
I20260927 22:51:16.509096 2017 consensus_queue.cc:260] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } }
I20260927 22:51:16.509634 1857 heartbeater.cc:503] Master 127.0.97.62:38563 was elected leader, sending a full tablet report...
I20260927 22:51:16.512192 2015 ts_tablet_manager.cc:1428] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260927 22:51:16.514437 1996 heartbeater.cc:503] Master 127.0.97.62:38563 was elected leader, sending a full tablet report...
I20260927 22:51:16.514946 2017 ts_tablet_manager.cc:1428] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: Time spent starting tablet: real 0.018s user 0.015s sys 0.003s
I20260927 22:51:16.589059 2021 raft_consensus.cc:482] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:16.589385 2021 raft_consensus.cc:504] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } }
I20260927 22:51:16.591012 2021 leader_election.cc:288] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 450c3c4c3ab94c11b044cee4780174e9 (127.0.97.3:42479), 4a670118694b4833a989e1e663a03a4c (127.0.97.2:37935)
I20260927 22:51:16.597977 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.591412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49779 (local address 127.0.97.2:37935)
0927 22:51:16.591582 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:16.591586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:16.592363 (+ 777us) server_negotiation.cc:407] Connection header received
0927 22:51:16.592600 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:16.592612 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:16.592669 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:16.592773 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:16.593234 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.594051 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.596229 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.596977 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.597143 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.597251 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:16.597566 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:51:16.597712 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:51:16.598086 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.591525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37935 (local address 127.0.97.1:49779)
0927 22:51:16.592231 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:16.592247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:16.592452 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:16.592782 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:16.592787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:16.592811 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:16.593102 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.593110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.594181 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.594186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.596078 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.596090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.597186 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.597192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.597357 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.597437 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 22:51:16.597542 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:51:16.597685 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":99,"thread_start_us":419,"threads_started":1}
I20260927 22:51:16.598093 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.591403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46297 (local address 127.0.97.3:42479)
0927 22:51:16.591597 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:16.591606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:16.592713 (+ 1107us) server_negotiation.cc:407] Connection header received
0927 22:51:16.592914 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:16.592919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:16.592980 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:16.593070 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:16.593644 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.594336 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.596386 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.597142 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.597298 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.597405 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:16.597720 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:51:16.597846 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:51:16.598208 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.591745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42479 (local address 127.0.97.1:46297)
0927 22:51:16.592603 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:16.592621 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:16.592783 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:16.593094 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:16.593098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:16.593121 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:16.593465 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.593473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.594474 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.594480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.596235 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:16.596245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.597358 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.597364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:16.597530 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.597551 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:51:16.597693 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:51:16.597827 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":3,"thread_start_us":561,"threads_started":1}
I20260927 22:51:16.599368 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad8a1f6dee54cd480c5dd5b25bba713" candidate_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a670118694b4833a989e1e663a03a4c" is_pre_election: true
I20260927 22:51:16.599370 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad8a1f6dee54cd480c5dd5b25bba713" candidate_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3c4c3ab94c11b044cee4780174e9" is_pre_election: true
I20260927 22:51:16.599841 1946 raft_consensus.cc:2454] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd0304cc23a4d57a31ee0dc4fbc7759 in term 0.
I20260927 22:51:16.599845 1812 raft_consensus.cc:2454] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd0304cc23a4d57a31ee0dc4fbc7759 in term 0.
I20260927 22:51:16.600497 1600 leader_election.cc:302] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 450c3c4c3ab94c11b044cee4780174e9, 4a670118694b4833a989e1e663a03a4c, ffd0304cc23a4d57a31ee0dc4fbc7759; no voters:
I20260927 22:51:16.600873 2021 raft_consensus.cc:2781] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:16.601022 2021 raft_consensus.cc:482] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:16.601125 2021 raft_consensus.cc:3037] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:16.603324 2021 raft_consensus.cc:504] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } }
I20260927 22:51:16.603938 2021 leader_election.cc:288] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [CANDIDATE]: Term 1 election: Requested vote from peers 450c3c4c3ab94c11b044cee4780174e9 (127.0.97.3:42479), 4a670118694b4833a989e1e663a03a4c (127.0.97.2:37935)
I20260927 22:51:16.604326 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad8a1f6dee54cd480c5dd5b25bba713" candidate_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3c4c3ab94c11b044cee4780174e9"
I20260927 22:51:16.604408 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad8a1f6dee54cd480c5dd5b25bba713" candidate_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a670118694b4833a989e1e663a03a4c"
I20260927 22:51:16.604501 1946 raft_consensus.cc:3037] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:16.604709 1812 raft_consensus.cc:3037] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:16.607194 1946 raft_consensus.cc:2454] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd0304cc23a4d57a31ee0dc4fbc7759 in term 1.
I20260927 22:51:16.607540 1600 leader_election.cc:302] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450c3c4c3ab94c11b044cee4780174e9, ffd0304cc23a4d57a31ee0dc4fbc7759; no voters:
I20260927 22:51:16.607577 1812 raft_consensus.cc:2454] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd0304cc23a4d57a31ee0dc4fbc7759 in term 1.
I20260927 22:51:16.607792 2021 raft_consensus.cc:2781] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:16.608775 2021 raft_consensus.cc:686] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 LEADER]: Becoming Leader. State: Replica: ffd0304cc23a4d57a31ee0dc4fbc7759, State: Running, Role: LEADER
W20260927 22:51:16.609376 1858 tablet.cc:2369] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:16.609328 2021 consensus_queue.cc:237] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } }
I20260927 22:51:16.613735 1511 catalog_manager.cc:5495] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffd0304cc23a4d57a31ee0dc4fbc7759 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } health_report { overall_health: UNKNOWN } } }
W20260927 22:51:16.623320 1997 tablet.cc:2369] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:51:16.627455 1719 tablet.cc:2369] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:16.660604 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:16.655117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.0.97.1:44279)
0927 22:51:16.655324 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:16.655330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:16.655486 (+ 156us) server_negotiation.cc:407] Connection header received
0927 22:51:16.655812 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:16.655816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:16.655911 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:16.656053 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:16.656423 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.657125 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.658509 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:16.658914 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:16.659050 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:16.659522 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:16.659989 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:16.660139 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:16.660271 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 22:51:16.660347 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:51:16.668363 1812 raft_consensus.cc:1261] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Refusing update from remote peer ffd0304cc23a4d57a31ee0dc4fbc7759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:16.668380 1946 raft_consensus.cc:1261] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Refusing update from remote peer ffd0304cc23a4d57a31ee0dc4fbc7759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:16.669384 2021 consensus_queue.cc:1035] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:16.669734 2026 consensus_queue.cc:1035] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:16.704543 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:16.704878 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:16.708417 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b: Sending DeleteTablet for 3 replicas of tablet 1ad8a1f6dee54cd480c5dd5b25bba713
I20260927 22:51:16.709555 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad8a1f6dee54cd480c5dd5b25bba713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52230
I20260927 22:51:16.709583 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad8a1f6dee54cd480c5dd5b25bba713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50742
I20260927 22:51:16.709685 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad8a1f6dee54cd480c5dd5b25bba713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56876
I20260927 22:51:16.712412 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:16.714120 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:16.724550 1647 tablet_service.cc:1467] Processing CreateTablet for tablet a456128efd9241f7a951df5a144d5c71 (DEFAULT_TABLE table=test-table [id=677ae8908cfe4f6f94afcb3ac33a93e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:16.725080 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a456128efd9241f7a951df5a144d5c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:16.725222 1925 tablet_service.cc:1467] Processing CreateTablet for tablet a456128efd9241f7a951df5a144d5c71 (DEFAULT_TABLE table=test-table [id=677ae8908cfe4f6f94afcb3ac33a93e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:16.725157 1786 tablet_service.cc:1467] Processing CreateTablet for tablet a456128efd9241f7a951df5a144d5c71 (DEFAULT_TABLE table=test-table [id=677ae8908cfe4f6f94afcb3ac33a93e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:16.725744 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a456128efd9241f7a951df5a144d5c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:16.725873 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a456128efd9241f7a951df5a144d5c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:16.731917 2016 tablet_bootstrap.cc:492] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759: Bootstrap starting.
I20260927 22:51:16.732185 2017 tablet_bootstrap.cc:492] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9: Bootstrap starting.
I20260927 22:51:16.733534 2016 tablet_bootstrap.cc:654] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:16.733726 2017 tablet_bootstrap.cc:654] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:16.737991 2015 tablet_bootstrap.cc:492] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c: Bootstrap starting.
I20260927 22:51:16.739982 2015 tablet_bootstrap.cc:654] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:16.744391 2040 tablet_replica.cc:330] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: stopping tablet replica
I20260927 22:51:16.745828 2015 tablet_bootstrap.cc:492] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c: No bootstrap required, opened a new log
I20260927 22:51:16.745987 2015 ts_tablet_manager.cc:1397] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260927 22:51:16.746122 2040 raft_consensus.cc:2229] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:16.746686 2040 raft_consensus.cc:2258] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:16.746785 2015 raft_consensus.cc:348] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:16.747035 2015 raft_consensus.cc:374] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:16.747128 2015 raft_consensus.cc:729] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a670118694b4833a989e1e663a03a4c, State: Initialized, Role: FOLLOWER
I20260927 22:51:16.747458 2016 tablet_bootstrap.cc:492] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759: No bootstrap required, opened a new log
I20260927 22:51:16.747371 2015 consensus_queue.cc:260] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:16.747648 2016 ts_tablet_manager.cc:1397] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260927 22:51:16.748490 2016 raft_consensus.cc:348] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:16.748667 2015 ts_tablet_manager.cc:1428] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 22:51:16.748780 2016 raft_consensus.cc:374] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:16.748903 2016 raft_consensus.cc:729] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd0304cc23a4d57a31ee0dc4fbc7759, State: Initialized, Role: FOLLOWER
I20260927 22:51:16.749214 2016 consensus_queue.cc:260] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:16.749929 2016 ts_tablet_manager.cc:1428] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:51:16.753353 2017 tablet_bootstrap.cc:492] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9: No bootstrap required, opened a new log
I20260927 22:51:16.753500 2017 ts_tablet_manager.cc:1397] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.009s
I20260927 22:51:16.754324 2017 raft_consensus.cc:348] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:16.754557 2017 raft_consensus.cc:374] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:16.754665 2017 raft_consensus.cc:729] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450c3c4c3ab94c11b044cee4780174e9, State: Initialized, Role: FOLLOWER
I20260927 22:51:16.755000 2017 consensus_queue.cc:260] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:16.755618 2017 ts_tablet_manager.cc:1428] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:51:16.757800 2038 tablet_replica.cc:330] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: stopping tablet replica
I20260927 22:51:16.758201 2040 ts_tablet_manager.cc:1905] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:16.763913 2038 raft_consensus.cc:2229] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:16.764461 2038 raft_consensus.cc:2258] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:16.765470 2039 tablet_replica.cc:330] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: stopping tablet replica
I20260927 22:51:16.765977 2039 raft_consensus.cc:2229] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:16.766671 2038 ts_tablet_manager.cc:1905] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:16.766696 2039 raft_consensus.cc:2258] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:16.766759 2040 ts_tablet_manager.cc:1918] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:16.766942 2040 log.cc:1196] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/1ad8a1f6dee54cd480c5dd5b25bba713
I20260927 22:51:16.767518 2040 ts_tablet_manager.cc:1939] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 4a670118694b4833a989e1e663a03a4c: Deleting consensus metadata
I20260927 22:51:16.769258 1498 catalog_manager.cc:4841] TS 4a670118694b4833a989e1e663a03a4c (127.0.97.2:37935): tablet 1ad8a1f6dee54cd480c5dd5b25bba713 (table test-table [id=7fe16411c9b7488f89e42fa85937328e]) successfully deleted
I20260927 22:51:16.770496 2039 ts_tablet_manager.cc:1905] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:16.772291 2038 ts_tablet_manager.cc:1918] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:16.772449 2038 log.cc:1196] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/1ad8a1f6dee54cd480c5dd5b25bba713
I20260927 22:51:16.773013 2038 ts_tablet_manager.cc:1939] T 1ad8a1f6dee54cd480c5dd5b25bba713 P 450c3c4c3ab94c11b044cee4780174e9: Deleting consensus metadata
I20260927 22:51:16.774500 1497 catalog_manager.cc:4841] TS 450c3c4c3ab94c11b044cee4780174e9 (127.0.97.3:42479): tablet 1ad8a1f6dee54cd480c5dd5b25bba713 (table test-table [id=7fe16411c9b7488f89e42fa85937328e]) successfully deleted
I20260927 22:51:16.777678 2039 ts_tablet_manager.cc:1918] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:16.777841 2039 log.cc:1196] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/1ad8a1f6dee54cd480c5dd5b25bba713
I20260927 22:51:16.778350 2039 ts_tablet_manager.cc:1939] T 1ad8a1f6dee54cd480c5dd5b25bba713 P ffd0304cc23a4d57a31ee0dc4fbc7759: Deleting consensus metadata
I20260927 22:51:16.779816 1497 catalog_manager.cc:4841] TS ffd0304cc23a4d57a31ee0dc4fbc7759 (127.0.97.1:44279): tablet 1ad8a1f6dee54cd480c5dd5b25bba713 (table test-table [id=7fe16411c9b7488f89e42fa85937328e]) successfully deleted
I20260927 22:51:17.119385 2023 raft_consensus.cc:482] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:17.119624 2023 raft_consensus.cc:504] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.121109 2023 leader_election.cc:288] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffd0304cc23a4d57a31ee0dc4fbc7759 (127.0.97.1:44279), 4a670118694b4833a989e1e663a03a4c (127.0.97.2:37935)
I20260927 22:51:17.121966 2026 raft_consensus.cc:482] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:17.122154 2026 raft_consensus.cc:504] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.122756 2026 leader_election.cc:288] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a670118694b4833a989e1e663a03a4c (127.0.97.2:37935), 450c3c4c3ab94c11b044cee4780174e9 (127.0.97.3:42479)
I20260927 22:51:17.123392 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456128efd9241f7a951df5a144d5c71" candidate_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a670118694b4833a989e1e663a03a4c" is_pre_election: true
I20260927 22:51:17.123685 1812 raft_consensus.cc:2454] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd0304cc23a4d57a31ee0dc4fbc7759 in term 0.
I20260927 22:51:17.124698 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456128efd9241f7a951df5a144d5c71" candidate_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3c4c3ab94c11b044cee4780174e9" is_pre_election: true
I20260927 22:51:17.124956 1946 raft_consensus.cc:2454] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd0304cc23a4d57a31ee0dc4fbc7759 in term 0.
I20260927 22:51:17.125710 1600 leader_election.cc:302] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450c3c4c3ab94c11b044cee4780174e9, ffd0304cc23a4d57a31ee0dc4fbc7759; no voters:
I20260927 22:51:17.126041 2026 raft_consensus.cc:2781] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:17.126173 2026 raft_consensus.cc:482] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:17.126272 2026 raft_consensus.cc:3037] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:17.130115 2026 raft_consensus.cc:504] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.131481 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456128efd9241f7a951df5a144d5c71" candidate_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a670118694b4833a989e1e663a03a4c"
I20260927 22:51:17.131747 1812 raft_consensus.cc:3037] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:17.135632 1812 raft_consensus.cc:2454] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd0304cc23a4d57a31ee0dc4fbc7759 in term 1.
I20260927 22:51:17.136219 2026 leader_election.cc:288] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [CANDIDATE]: Term 1 election: Requested vote from peers 4a670118694b4833a989e1e663a03a4c (127.0.97.2:37935), 450c3c4c3ab94c11b044cee4780174e9 (127.0.97.3:42479)
I20260927 22:51:17.136242 1601 leader_election.cc:302] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a670118694b4833a989e1e663a03a4c, ffd0304cc23a4d57a31ee0dc4fbc7759; no voters:
I20260927 22:51:17.136518 2026 raft_consensus.cc:2781] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:17.136636 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456128efd9241f7a951df5a144d5c71" candidate_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3c4c3ab94c11b044cee4780174e9"
I20260927 22:51:17.136761 2026 raft_consensus.cc:686] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 LEADER]: Becoming Leader. State: Replica: ffd0304cc23a4d57a31ee0dc4fbc7759, State: Running, Role: LEADER
I20260927 22:51:17.136871 1946 raft_consensus.cc:3037] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:17.137019 2026 consensus_queue.cc:237] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.139884 1946 raft_consensus.cc:2454] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd0304cc23a4d57a31ee0dc4fbc7759 in term 1.
I20260927 22:51:17.142339 1510 catalog_manager.cc:5495] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffd0304cc23a4d57a31ee0dc4fbc7759 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } health_report { overall_health: UNKNOWN } } }
I20260927 22:51:17.143673 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.122290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56469 (local address 127.0.97.2:37935)
0927 22:51:17.135599 (+ 13309us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.135606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.136333 (+ 727us) server_negotiation.cc:407] Connection header received
0927 22:51:17.136396 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.136401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.136495 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.136597 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:17.137208 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.138330 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.141516 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.142697 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.142815 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.142900 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.143247 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.143376 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13168,"thread_start_us":13007,"threads_started":1}
I20260927 22:51:17.143743 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.121434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37935 (local address 127.0.97.3:56469)
0927 22:51:17.136107 (+ 14673us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:17.136125 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:17.136233 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:17.136655 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:17.136660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:17.136691 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:17.136933 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.136945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.138495 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.138500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.141337 (+ 2837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.141352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.142861 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.142869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.143048 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.143084 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:51:17.143195 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:51:17.143328 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14558,"mutex_wait_us":1,"thread_start_us":14102,"threads_started":1}
I20260927 22:51:17.144377 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.121781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34687 (local address 127.0.97.1:44279)
0927 22:51:17.122062 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.122071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.135898 (+ 13827us) server_negotiation.cc:407] Connection header received
0927 22:51:17.137179 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.137184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.137254 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.137321 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:17.138169 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.139610 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.142286 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.143284 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.143398 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.143496 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.143988 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.144121 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260927 22:51:17.144546 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456128efd9241f7a951df5a144d5c71" candidate_uuid: "450c3c4c3ab94c11b044cee4780174e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a670118694b4833a989e1e663a03a4c" is_pre_election: true
I20260927 22:51:17.144847 1812 raft_consensus.cc:2379] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 450c3c4c3ab94c11b044cee4780174e9 in current term 1: Already voted for candidate ffd0304cc23a4d57a31ee0dc4fbc7759 in this term.
I20260927 22:51:17.145013 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.121406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44279 (local address 127.0.97.3:34687)
0927 22:51:17.135782 (+ 14376us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:17.135808 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:17.136018 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:17.137513 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:17.137518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:17.137541 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:17.137916 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.137928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.139850 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.139859 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.142142 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.142153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.143603 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.143610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.143822 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.143852 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:51:17.144524 (+ 672us) client_negotiation.cc:240] Negotiation successful
0927 22:51:17.144701 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14227,"mutex_wait_us":1,"thread_start_us":14430,"threads_started":1}
I20260927 22:51:17.145958 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456128efd9241f7a951df5a144d5c71" candidate_uuid: "450c3c4c3ab94c11b044cee4780174e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" is_pre_election: true
I20260927 22:51:17.146685 1879 leader_election.cc:302] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 450c3c4c3ab94c11b044cee4780174e9; no voters: 4a670118694b4833a989e1e663a03a4c, ffd0304cc23a4d57a31ee0dc4fbc7759
I20260927 22:51:17.147040 2023 raft_consensus.cc:2726] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 27 22:51:17 dist-test-slave-df53 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:17.241489 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.227919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38563 (local address 127.0.0.1:48240)
0927 22:51:17.228692 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:17.228709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:17.228876 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:17.229275 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:17.229281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:17.230011 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:17.230188 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.230197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.231223 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.231227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.232223 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.232234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.232906 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.232910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.233032 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.233679 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:17.233702 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:17.236318 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:17.239340 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:17.239348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:17.239353 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:17.239755 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:17.240098 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:17.240100 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:17.240103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:17.240229 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:17.240708 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:17.240714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:17.240850 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 22:51:17.240980 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 22:51:17.241167 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":498,"threads_started":1}
I20260927 22:51:17.241581 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.227893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.0.97.62:38563)
0927 22:51:17.228544 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.228556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.228784 (+ 228us) server_negotiation.cc:407] Connection header received
0927 22:51:17.229013 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.229017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.229088 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.229186 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:17.230325 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.231075 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.232345 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.232756 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.232844 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.236512 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:17.236517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:17.236520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:17.236546 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:17.239211 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.239879 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.239883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.239885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.240001 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.240320 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.240323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.240325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.240566 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:17.240676 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.241062 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.241293 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":504,"thread_start_us":321,"threads_started":1}
I20260927 22:51:17.246037 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48240:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:51:17.247018 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:17.253562 1648 tablet_service.cc:1467] Processing CreateTablet for tablet ce2e9e741ef94debaeb54dd9ec270a12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64f591b4809b450bb7e84ba491025d5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:51:17.253614 1787 tablet_service.cc:1467] Processing CreateTablet for tablet ce2e9e741ef94debaeb54dd9ec270a12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64f591b4809b450bb7e84ba491025d5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:51:17.254127 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce2e9e741ef94debaeb54dd9ec270a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:17.253985 1926 tablet_service.cc:1467] Processing CreateTablet for tablet ce2e9e741ef94debaeb54dd9ec270a12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64f591b4809b450bb7e84ba491025d5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:51:17.254153 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce2e9e741ef94debaeb54dd9ec270a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:17.254462 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce2e9e741ef94debaeb54dd9ec270a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:17.259842 2055 tablet_bootstrap.cc:492] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: Bootstrap starting.
I20260927 22:51:17.261180 2056 tablet_bootstrap.cc:492] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: Bootstrap starting.
I20260927 22:51:17.261263 2055 tablet_bootstrap.cc:654] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:17.261912 2057 tablet_bootstrap.cc:492] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: Bootstrap starting.
I20260927 22:51:17.263293 2056 tablet_bootstrap.cc:654] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:17.263448 2057 tablet_bootstrap.cc:654] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:17.264174 2055 tablet_bootstrap.cc:492] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: No bootstrap required, opened a new log
I20260927 22:51:17.264362 2055 ts_tablet_manager.cc:1397] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 22:51:17.265008 2055 raft_consensus.cc:348] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.265241 2055 raft_consensus.cc:374] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:17.265367 2055 raft_consensus.cc:729] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd0304cc23a4d57a31ee0dc4fbc7759, State: Initialized, Role: FOLLOWER
I20260927 22:51:17.265434 2056 tablet_bootstrap.cc:492] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: No bootstrap required, opened a new log
I20260927 22:51:17.265558 2056 ts_tablet_manager.cc:1397] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:51:17.265677 2055 consensus_queue.cc:260] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.266381 2056 raft_consensus.cc:348] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.266427 2026 tablet_replica.cc:439] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } }
I20260927 22:51:17.266659 2056 raft_consensus.cc:374] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:17.266672 2026 tablet_replica.cc:442] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:17.266752 2056 raft_consensus.cc:729] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450c3c4c3ab94c11b044cee4780174e9, State: Initialized, Role: FOLLOWER
I20260927 22:51:17.267052 2057 tablet_bootstrap.cc:492] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: No bootstrap required, opened a new log
I20260927 22:51:17.266963 2056 consensus_queue.cc:260] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.267150 2057 ts_tablet_manager.cc:1397] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 22:51:17.267405 2023 tablet_replica.cc:439] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } }
I20260927 22:51:17.267549 2056 ts_tablet_manager.cc:1428] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:51:17.267642 2023 tablet_replica.cc:442] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:17.267848 2057 raft_consensus.cc:348] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.268057 2057 raft_consensus.cc:374] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:17.268152 2057 raft_consensus.cc:729] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a670118694b4833a989e1e663a03a4c, State: Initialized, Role: FOLLOWER
I20260927 22:51:17.268376 2057 consensus_queue.cc:260] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.269758 2055 ts_tablet_manager.cc:1428] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 22:51:17.269888 2057 ts_tablet_manager.cc:1428] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 22:51:17.269654 2061 tablet_replica.cc:439] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } }
I20260927 22:51:17.270596 2061 tablet_replica.cc:442] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:17.400887 2061 raft_consensus.cc:482] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:17.401070 2061 raft_consensus.cc:504] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.402580 2061 leader_election.cc:288] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffd0304cc23a4d57a31ee0dc4fbc7759 (127.0.97.1:44279), 450c3c4c3ab94c11b044cee4780174e9 (127.0.97.3:42479)
I20260927 22:51:17.409583 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.402958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44279 (local address 127.0.97.2:42765)
0927 22:51:17.403411 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:17.403427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:17.403604 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:17.404105 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:17.404115 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:17.404142 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:17.404500 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.404509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.405872 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.405878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.407624 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.407639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.408913 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.408946 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.409102 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.409128 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:51:17.409213 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:51:17.409308 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":3,"thread_start_us":225,"threads_started":1}
I20260927 22:51:17.409693 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.403046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42765 (local address 127.0.97.1:44279)
0927 22:51:17.403267 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.403274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.403526 (+ 252us) server_negotiation.cc:407] Connection header received
0927 22:51:17.403735 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.403740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.403841 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.403938 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:17.404641 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.405687 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.407797 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.408687 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.408816 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.408911 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.409267 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.409415 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:51:17.410264 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2e9e741ef94debaeb54dd9ec270a12" candidate_uuid: "4a670118694b4833a989e1e663a03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" is_pre_election: true
I20260927 22:51:17.410570 1672 raft_consensus.cc:2454] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a670118694b4833a989e1e663a03a4c in term 0.
I20260927 22:51:17.410861 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.403554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42479 (local address 127.0.97.2:55975)
0927 22:51:17.404122 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:17.404137 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:17.404244 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:17.404808 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:17.404814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:17.404841 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:17.405262 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.405275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.406674 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.406679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.408336 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.408347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.410099 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.410105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.410254 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.410275 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:51:17.410383 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:51:17.410506 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":290,"threads_started":1}
I20260927 22:51:17.410871 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.403635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55975 (local address 127.0.97.3:42479)
0927 22:51:17.404396 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.404400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.404418 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:51:17.404472 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.404476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.404572 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.404688 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:17.405564 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.406542 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.408495 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.409705 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.409836 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.409963 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.410410 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.410542 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":505,"threads_started":1}
I20260927 22:51:17.411078 1740 leader_election.cc:302] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a670118694b4833a989e1e663a03a4c, ffd0304cc23a4d57a31ee0dc4fbc7759; no voters:
I20260927 22:51:17.411471 2061 raft_consensus.cc:2781] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:17.411604 2061 raft_consensus.cc:482] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:17.411722 2061 raft_consensus.cc:3037] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:17.411939 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2e9e741ef94debaeb54dd9ec270a12" candidate_uuid: "4a670118694b4833a989e1e663a03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3c4c3ab94c11b044cee4780174e9" is_pre_election: true
I20260927 22:51:17.412643 1946 raft_consensus.cc:2454] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a670118694b4833a989e1e663a03a4c in term 0.
I20260927 22:51:17.414402 2061 raft_consensus.cc:504] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.414867 2061 leader_election.cc:288] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [CANDIDATE]: Term 1 election: Requested vote from peers ffd0304cc23a4d57a31ee0dc4fbc7759 (127.0.97.1:44279), 450c3c4c3ab94c11b044cee4780174e9 (127.0.97.3:42479)
I20260927 22:51:17.415320 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2e9e741ef94debaeb54dd9ec270a12" candidate_uuid: "4a670118694b4833a989e1e663a03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759"
I20260927 22:51:17.415443 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2e9e741ef94debaeb54dd9ec270a12" candidate_uuid: "4a670118694b4833a989e1e663a03a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450c3c4c3ab94c11b044cee4780174e9"
I20260927 22:51:17.415560 1672 raft_consensus.cc:3037] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:17.415725 1946 raft_consensus.cc:3037] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:17.417557 1672 raft_consensus.cc:2454] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a670118694b4833a989e1e663a03a4c in term 1.
I20260927 22:51:17.417577 1946 raft_consensus.cc:2454] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a670118694b4833a989e1e663a03a4c in term 1.
I20260927 22:51:17.418071 1740 leader_election.cc:302] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a670118694b4833a989e1e663a03a4c, ffd0304cc23a4d57a31ee0dc4fbc7759; no voters:
I20260927 22:51:17.418358 2061 raft_consensus.cc:2781] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:17.418812 2061 raft_consensus.cc:686] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [term 1 LEADER]: Becoming Leader. State: Replica: 4a670118694b4833a989e1e663a03a4c, State: Running, Role: LEADER
I20260927 22:51:17.419166 2061 consensus_queue.cc:237] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } }
I20260927 22:51:17.421012 2065 tablet_replica.cc:439] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: TxnStatusTablet state changed. Reason: New leader 4a670118694b4833a989e1e663a03a4c. Latest consensus state: current_term: 1 leader_uuid: "4a670118694b4833a989e1e663a03a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } }
I20260927 22:51:17.421391 2065 tablet_replica.cc:442] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:51:17.422168 2067 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:51:17.422374 2067 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:51:17.422668 1511 catalog_manager.cc:5495] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a670118694b4833a989e1e663a03a4c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4a670118694b4833a989e1e663a03a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } health_report { overall_health: UNKNOWN } } }
I20260927 22:51:17.456971 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.450464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37935 (local address 127.0.0.1:52232)
0927 22:51:17.450707 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:17.450734 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:17.450855 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:17.451361 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:17.451364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:17.451373 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:51:17.451515 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.451522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.452884 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.452888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.454390 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.454402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.455331 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.455336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.455482 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.455553 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:51:17.456555 (+ 1002us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:51:17.456570 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 22:51:17.456622 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 22:51:17.456739 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 22:51:17.457024 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.450561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.0.97.2:37935)
0927 22:51:17.450851 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.450857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.450878 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:51:17.451049 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.451055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.451189 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.451341 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:17.451649 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.452760 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.454525 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.455174 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.455302 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.455754 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:17.456446 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:17.456595 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.456653 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.456745 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260927 22:51:17.461396 1946 raft_consensus.cc:1261] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Refusing update from remote peer 4a670118694b4833a989e1e663a03a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:17.461582 1672 raft_consensus.cc:1261] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 FOLLOWER]: Refusing update from remote peer 4a670118694b4833a989e1e663a03a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:17.462164 2061 consensus_queue.cc:1035] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:17.462509 2065 consensus_queue.cc:1035] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:17.471351 2023 tablet_replica.cc:439] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: TxnStatusTablet state changed. Reason: New leader 4a670118694b4833a989e1e663a03a4c. Latest consensus state: current_term: 1 leader_uuid: "4a670118694b4833a989e1e663a03a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } }
I20260927 22:51:17.471683 2023 tablet_replica.cc:442] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:17.473076 2026 tablet_replica.cc:439] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: TxnStatusTablet state changed. Reason: New leader 4a670118694b4833a989e1e663a03a4c. Latest consensus state: current_term: 1 leader_uuid: "4a670118694b4833a989e1e663a03a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } }
I20260927 22:51:17.473346 2026 tablet_replica.cc:442] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:17.474757 2070 mvcc.cc:205] Tried to move back new op lower bound from 7334090659675107328 to 7334090659512455168. Current Snapshot: MvccSnapshot[applied={T|T < 7334090659675107328}]
I20260927 22:51:17.475661 2071 mvcc.cc:205] Tried to move back new op lower bound from 7334090659675107328 to 7334090659512455168. Current Snapshot: MvccSnapshot[applied={T|T < 7334090659675107328}]
I20260927 22:51:17.476230 2065 tablet_replica.cc:439] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a670118694b4833a989e1e663a03a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } }
I20260927 22:51:17.476539 2065 tablet_replica.cc:442] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:51:17.481899 2023 tablet_replica.cc:439] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a670118694b4833a989e1e663a03a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } }
I20260927 22:51:17.482255 2023 tablet_replica.cc:442] T ce2e9e741ef94debaeb54dd9ec270a12 P 450c3c4c3ab94c11b044cee4780174e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:17.488997 2026 tablet_replica.cc:439] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a670118694b4833a989e1e663a03a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } }
I20260927 22:51:17.489256 2026 tablet_replica.cc:442] T ce2e9e741ef94debaeb54dd9ec270a12 P ffd0304cc23a4d57a31ee0dc4fbc7759: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:17.491103 2077 tablet_replica.cc:439] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a670118694b4833a989e1e663a03a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd0304cc23a4d57a31ee0dc4fbc7759" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44279 } } peers { permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 } } peers { permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 } } }
I20260927 22:51:17.491590 2077 tablet_replica.cc:442] T ce2e9e741ef94debaeb54dd9ec270a12 P 4a670118694b4833a989e1e663a03a4c: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:51:17.501456 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.495264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43117 (local address 127.0.97.2:37935)
0927 22:51:17.495453 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.495460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.496073 (+ 613us) server_negotiation.cc:407] Connection header received
0927 22:51:17.496247 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.496251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.496309 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.496389 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:17.496949 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.497813 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.499774 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.500182 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.500275 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.500803 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:17.500939 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:17.501031 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.501184 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.501243 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:51:17.501773 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.495292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37935 (local address 127.0.97.1:43117)
0927 22:51:17.495964 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:17.495985 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:17.496125 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:17.496428 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:17.496433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:17.496446 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:51:17.496807 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.496818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.497938 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.497944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.499621 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.499637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.500330 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.500337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.500514 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.500594 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:51:17.501049 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:51:17.501061 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:51:17.501144 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 22:51:17.501279 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":338,"threads_started":1}
I20260927 22:51:17.511457 1946 raft_consensus.cc:1261] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Refusing update from remote peer ffd0304cc23a4d57a31ee0dc4fbc7759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:17.511523 1812 raft_consensus.cc:1261] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Refusing update from remote peer ffd0304cc23a4d57a31ee0dc4fbc7759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:17.512068 2021 consensus_queue.cc:1035] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a670118694b4833a989e1e663a03a4c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:17.512398 2026 consensus_queue.cc:1035] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450c3c4c3ab94c11b044cee4780174e9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:17.521895 2072 mvcc.cc:205] Tried to move back new op lower bound from 7334090659881938944 to 7334090658356318208. Current Snapshot: MvccSnapshot[applied={T|T < 7334090659881938944}]
I20260927 22:51:17.552386 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.544681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40705 (local address 127.0.97.1:44279)
0927 22:51:17.544866 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.544872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.545254 (+ 382us) server_negotiation.cc:407] Connection header received
0927 22:51:17.545540 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.545544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.545689 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.545800 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:17.546658 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.547592 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.550268 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.550775 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.550930 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.551529 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:17.551704 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:17.551810 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.551948 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.552028 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:51:17.552533 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.544304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44279 (local address 127.0.97.2:40705)
0927 22:51:17.545145 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:17.545166 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:17.545375 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:17.545982 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:17.545986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:17.546009 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:51:17.546483 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.546499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.547725 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.547733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.550124 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:17.550136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.550967 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.550976 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:17.551232 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.551319 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:51:17.551810 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:51:17.551824 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:51:17.551928 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:51:17.552071 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":746,"threads_started":1}
I20260927 22:51:17.595041 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:17.595289 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:17.598207 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5265b9ff0c084b8ea87abd1f5331921b: Sending DeleteTablet for 3 replicas of tablet a456128efd9241f7a951df5a144d5c71
I20260927 22:51:17.598987 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet a456128efd9241f7a951df5a144d5c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52230
I20260927 22:51:17.599207 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet a456128efd9241f7a951df5a144d5c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50742
I20260927 22:51:17.599607 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet a456128efd9241f7a951df5a144d5c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56876
I20260927 22:51:17.607529 2089 tablet_replica.cc:330] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9: stopping tablet replica
I20260927 22:51:17.607811 2089 raft_consensus.cc:2229] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:17.608116 2089 raft_consensus.cc:2258] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:17.609061 2088 tablet_replica.cc:330] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c: stopping tablet replica
I20260927 22:51:17.609318 2088 raft_consensus.cc:2229] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:17.609433 2090 tablet_replica.cc:330] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759: stopping tablet replica
I20260927 22:51:17.609701 2088 raft_consensus.cc:2258] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:17.609786 2090 raft_consensus.cc:2229] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:17.610430 2090 raft_consensus.cc:2258] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:17.611096 2088 ts_tablet_manager.cc:1905] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:17.612136 2090 ts_tablet_manager.cc:1905] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:17.613646 2089 ts_tablet_manager.cc:1905] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:17.618532 2088 ts_tablet_manager.cc:1918] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:51:17.618710 2088 log.cc:1196] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/a456128efd9241f7a951df5a144d5c71
I20260927 22:51:17.619221 2088 ts_tablet_manager.cc:1939] T a456128efd9241f7a951df5a144d5c71 P 4a670118694b4833a989e1e663a03a4c: Deleting consensus metadata
I20260927 22:51:17.620366 2089 ts_tablet_manager.cc:1918] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:51:17.620375 1498 catalog_manager.cc:4841] TS 4a670118694b4833a989e1e663a03a4c (127.0.97.2:37935): tablet a456128efd9241f7a951df5a144d5c71 (table test-table [id=677ae8908cfe4f6f94afcb3ac33a93e0]) successfully deleted
I20260927 22:51:17.621479 2089 log.cc:1196] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/a456128efd9241f7a951df5a144d5c71
I20260927 22:51:17.622473 2089 ts_tablet_manager.cc:1939] T a456128efd9241f7a951df5a144d5c71 P 450c3c4c3ab94c11b044cee4780174e9: Deleting consensus metadata
I20260927 22:51:17.623556 1497 catalog_manager.cc:4841] TS 450c3c4c3ab94c11b044cee4780174e9 (127.0.97.3:42479): tablet a456128efd9241f7a951df5a144d5c71 (table test-table [id=677ae8908cfe4f6f94afcb3ac33a93e0]) successfully deleted
I20260927 22:51:17.626242 2090 ts_tablet_manager.cc:1918] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:51:17.626369 2090 log.cc:1196] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/a456128efd9241f7a951df5a144d5c71
I20260927 22:51:17.626837 2090 ts_tablet_manager.cc:1939] T a456128efd9241f7a951df5a144d5c71 P ffd0304cc23a4d57a31ee0dc4fbc7759: Deleting consensus metadata
I20260927 22:51:17.627863 1497 catalog_manager.cc:4841] TS ffd0304cc23a4d57a31ee0dc4fbc7759 (127.0.97.1:44279): tablet a456128efd9241f7a951df5a144d5c71 (table test-table [id=677ae8908cfe4f6f94afcb3ac33a93e0]) successfully deleted
Sep 27 22:51:17 dist-test-slave-df53 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:17.644984 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.609788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.0.97.1:44279)
0927 22:51:17.609954 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.609960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.621233 (+ 11273us) server_negotiation.cc:407] Connection header received
0927 22:51:17.621408 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.621414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.621492 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.621619 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:17.623558 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.624851 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.626813 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.627463 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.627580 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.638695 (+ 11115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:17.638712 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:17.638734 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:17.638777 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:17.642241 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.642864 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.642875 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.642881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.643021 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.643343 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.643348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.643352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.643805 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:17.644088 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.644405 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.644613 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 22:51:17.645865 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56894
I20260927 22:51:17.661640 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.649222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.0.97.62:38563)
0927 22:51:17.649421 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.649427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.649676 (+ 249us) server_negotiation.cc:407] Connection header received
0927 22:51:17.649874 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.649880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.649955 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.650059 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:17.651678 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.652401 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.653573 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.654047 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.654189 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.656396 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:17.656402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:17.656407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:17.656446 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:17.659050 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.659725 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.659732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.659735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.659865 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.660208 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.660214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.660217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.660496 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:17.660760 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.661063 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.661278 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 22:51:17.662621 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48254
I20260927 22:51:17.664871 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 1582
I20260927 22:51:17.678954 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.666024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.0.97.1:44279)
0927 22:51:17.666225 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.666232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.666252 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:51:17.666403 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.666409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.666481 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.666586 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:17.667650 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.668765 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.669993 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.670707 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.670836 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.673077 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:17.673083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:17.673088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:17.673125 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:17.676543 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.677132 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.677138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.677141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.677268 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.677562 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.677567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.677570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.677903 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:17.678154 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.678379 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.678595 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 22:51:17.679581 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56896
W20260927 22:51:17.680387 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:51:17.680576 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 1582
I20260927 22:51:17.702010 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 1722
Sep 27 22:51:17 dist-test-slave-df53 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:17.718398 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.702723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.0.97.2:37935)
0927 22:51:17.702917 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.702922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.702943 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:51:17.703058 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.703064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.703164 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.703282 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:17.704482 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.705559 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.706875 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.707638 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.707783 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.711818 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:17.711832 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:17.711850 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:17.711895 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:17.715781 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.716370 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.716379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.716383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.716538 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.716879 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.716883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.716886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.717336 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:17.717652 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.717828 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.718048 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 22:51:17.719113 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52242
W20260927 22:51:17.719781 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:51:17.719944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 1722
I20260927 22:51:17.738714 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 1861
Sep 27 22:51:17 dist-test-slave-df53 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549476, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:51:17.753688 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:17.739440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.0.97.3:42479)
0927 22:51:17.739590 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:17.739594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:17.739612 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:51:17.739701 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:17.739705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:17.739771 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:17.739852 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:17.740956 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.741676 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.742994 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:17.743400 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:17.743487 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:17.748113 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:17.748126 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:17.748142 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:17.748181 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:17.750896 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.751537 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.751544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.751547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.751652 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:17.751981 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:17.751984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:17.751986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:17.752359 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:17.752694 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:17.753063 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 22:51:17.753301 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 22:51:17.754364 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50756
W20260927 22:51:17.754952 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:51:17.755123 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 1861
I20260927 22:51:17.771191 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 1479
I20260927 22:51:17.772302 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:17.884439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 1479
2026-09-27T22:51:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4773 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:17 dist-test-slave-df53 krb5kdc[2099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:17 dist-test-slave-df53 krb5kdc[2099](info): set up 2 sockets
Sep 27 22:51:17 dist-test-slave-df53 krb5kdc[2099](info): commencing operation
krb5kdc: starting...
W20260927 22:51:18.953545 388 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:19 dist-test-slave-df53 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:19.129040 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35035
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:19.434880 2112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:19.435209 2112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:19.435362 2112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:19.442597 2112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:19.442730 2112 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:51:19.442821 2112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:19.442939 2112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:19.443022 2112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:19.443112 2112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:19.452845 2112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35035
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:19.454144 2112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:19.455448 2112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:19.466974 2117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:19.467110 2118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:19.467144 2120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:19.468453 2112 server_base.cc:1034] running on GCE node
I20260927 22:51:19.469237 2112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:19.470569 2112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:19.471777 2112 hybrid_clock.cc:648] HybridClock initialized: now 1790549479471705 us; error 48 us; skew 500 ppm
I20260927 22:51:19.472101 2112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:19 dist-test-slave-df53 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:19.475211 2112 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:19.476509 2112 webserver.cc:460] Webserver started at http://127.0.97.62:44379/ using document root <none> and password file <none>
I20260927 22:51:19.477126 2112 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:19.477206 2112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:19.477437 2112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:19.480243 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "6af96a0326194af48ad02190f652a78f"
format_stamp: "Formatted at 2026-09-27 22:51:19 on dist-test-slave-df53"
server_key: "9b7cdb087e63cb0d02442731b7b089ef"
server_key_iv: "95ff9160487ab865558e3a6a23ff64fa"
server_key_version: "encryptionkey@0"
I20260927 22:51:19.480943 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "6af96a0326194af48ad02190f652a78f"
format_stamp: "Formatted at 2026-09-27 22:51:19 on dist-test-slave-df53"
server_key: "9b7cdb087e63cb0d02442731b7b089ef"
server_key_iv: "95ff9160487ab865558e3a6a23ff64fa"
server_key_version: "encryptionkey@0"
I20260927 22:51:19.485813 2112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260927 22:51:19.488926 2127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:19.490233 2112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:19.490404 2112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "6af96a0326194af48ad02190f652a78f"
format_stamp: "Formatted at 2026-09-27 22:51:19 on dist-test-slave-df53"
server_key: "9b7cdb087e63cb0d02442731b7b089ef"
server_key_iv: "95ff9160487ab865558e3a6a23ff64fa"
server_key_version: "encryptionkey@0"
I20260927 22:51:19.490605 2112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:19.503517 2112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:19.507675 2112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:19.507979 2112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:19.528976 2112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35035
I20260927 22:51:19.528986 2188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35035 every 8 connection(s)
I20260927 22:51:19.530283 2112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:19.533849 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 2112
I20260927 22:51:19.534050 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:19.534183 2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:19.534870 388 external_mini_cluster.cc:1428] Setting key b156f1225449e127286e0d1b9d9aa3c5
I20260927 22:51:19.547334 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f: Bootstrap starting.
Sep 27 22:51:19 dist-test-slave-df53 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:19.551062 2189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:19.552098 2189 log.cc:824] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:19.554836 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f: No bootstrap required, opened a new log
I20260927 22:51:19.558777 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:19.536866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.0.97.62:35035)
0927 22:51:19.537719 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:19.537730 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:19.537770 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:51:19.540088 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:19.540108 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:19.540487 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:19.540800 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:19.542494 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:19.543787 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:19.545577 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:19.546322 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:19.546455 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:19.551273 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:19.551281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:19.551295 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:19.551332 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:19.554418 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:19.555105 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:19.555117 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:19.555125 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:19.555362 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:19.555702 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:19.555707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:19.555710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:19.556343 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:19.556617 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:19.556990 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 22:51:19.557434 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":399,"threads_started":1}
I20260927 22:51:19.563383 2189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af96a0326194af48ad02190f652a78f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35035 } }
I20260927 22:51:19.563691 2189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:19.563764 2189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af96a0326194af48ad02190f652a78f, State: Initialized, Role: FOLLOWER
I20260927 22:51:19.564199 2189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af96a0326194af48ad02190f652a78f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35035 } }
I20260927 22:51:19.564383 2189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:19.564476 2189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:19.564595 2189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:19.566632 2189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af96a0326194af48ad02190f652a78f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35035 } }
I20260927 22:51:19.567130 2189 leader_election.cc:302] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6af96a0326194af48ad02190f652a78f; no voters:
I20260927 22:51:19.567785 2189 leader_election.cc:288] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:19.567886 2194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:19.568614 2194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [term 1 LEADER]: Becoming Leader. State: Replica: 6af96a0326194af48ad02190f652a78f, State: Running, Role: LEADER
I20260927 22:51:19.569088 2194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af96a0326194af48ad02190f652a78f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35035 } }
I20260927 22:51:19.569887 2189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:19.571921 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6af96a0326194af48ad02190f652a78f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af96a0326194af48ad02190f652a78f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35035 } } }
I20260927 22:51:19.571967 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6af96a0326194af48ad02190f652a78f. Latest consensus state: current_term: 1 leader_uuid: "6af96a0326194af48ad02190f652a78f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6af96a0326194af48ad02190f652a78f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35035 } } }
I20260927 22:51:19.572485 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:19.572984 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:19.574229 2199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:19.579859 2199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:19.588522 2199 catalog_manager.cc:1347] Generated new cluster ID: 8de1301920c0402dbd5412ab61875f84
I20260927 22:51:19.588647 2199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:19.597776 2199 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:19.599465 2199 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:19.618237 2199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f: Generated new TSK 0
I20260927 22:51:19.619186 2199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:19.661581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35035
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:19.975075 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:19.975348 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:19.975522 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:19.982815 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:19.982923 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:19.983006 2215 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:51:19.983057 2215 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:51:19.983116 2215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:19.992465 2215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:19.993745 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:19.995209 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:20.008260 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:20.008297 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:20.008320 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:20.009107 2215 server_base.cc:1034] running on GCE node
I20260927 22:51:20.009773 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:20.010663 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:20.011857 2215 hybrid_clock.cc:648] HybridClock initialized: now 1790549480011793 us; error 60 us; skew 500 ppm
I20260927 22:51:20.012167 2215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:20 dist-test-slave-df53 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:20.015321 2215 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:20.016731 2215 webserver.cc:460] Webserver started at http://127.0.97.1:33585/ using document root <none> and password file <none>
I20260927 22:51:20.017381 2215 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:20.017515 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:20.017853 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:20.020635 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "4f76d17dbf9945388d5d9d18f5bf1dea"
format_stamp: "Formatted at 2026-09-27 22:51:20 on dist-test-slave-df53"
server_key: "c5808f74390da4648ed8a4b20a4ae37b"
server_key_iv: "3994d9842bfb61c16eff5cae7d1e3df6"
server_key_version: "encryptionkey@0"
I20260927 22:51:20.021459 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4f76d17dbf9945388d5d9d18f5bf1dea"
format_stamp: "Formatted at 2026-09-27 22:51:20 on dist-test-slave-df53"
server_key: "c5808f74390da4648ed8a4b20a4ae37b"
server_key_iv: "3994d9842bfb61c16eff5cae7d1e3df6"
server_key_version: "encryptionkey@0"
I20260927 22:51:20.026413 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:51:20.029675 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:20.030978 2215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:20.031150 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "4f76d17dbf9945388d5d9d18f5bf1dea"
format_stamp: "Formatted at 2026-09-27 22:51:20 on dist-test-slave-df53"
server_key: "c5808f74390da4648ed8a4b20a4ae37b"
server_key_iv: "3994d9842bfb61c16eff5cae7d1e3df6"
server_key_version: "encryptionkey@0"
I20260927 22:51:20.031347 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:20.045943 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:20.049793 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:20.050072 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:20.052549 2215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:20.052685 2215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:20.052809 2215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:20.052904 2215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:51:20 dist-test-slave-df53 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:20.073823 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:20.053249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54183 (local address 127.0.97.62:35035)
0927 22:51:20.053489 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:20.053506 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:20.054830 (+ 1324us) server_negotiation.cc:407] Connection header received
0927 22:51:20.056758 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:20.056765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:20.056881 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:20.057058 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:20.059244 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.060264 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.062872 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.063294 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.063421 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:20.067001 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:20.067008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:20.067012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:20.067052 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:20.070237 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:20.071200 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:20.071205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:20.071208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:20.071393 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:20.071846 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:20.071851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:20.071855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:20.072158 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:20.072312 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:20.073265 (+ 953us) server_negotiation.cc:299] Negotiation successful
0927 22:51:20.073476 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:51:20.074908 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:20.053818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35035 (local address 127.0.97.1:54183)
0927 22:51:20.054649 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:20.054694 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:20.056500 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:20.057330 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:20.057341 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:20.058048 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:20.058986 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:20.059015 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.060420 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.060426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:20.062738 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:20.062749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.063532 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.063539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:20.063748 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:20.064376 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:20.064396 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:20.066741 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:20.070374 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:20.070382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:20.070397 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:20.071045 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:20.071515 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:20.071520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:20.071524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:20.071722 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:20.072393 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:20.072402 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:20.072802 (+ 400us) client_negotiation.cc:769] Sending connection context
0927 22:51:20.073178 (+ 376us) client_negotiation.cc:240] Negotiation successful
0927 22:51:20.073702 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":361,"threads_started":1}
I20260927 22:51:20.094372 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:20.080612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34651 (local address 127.0.97.62:35035)
0927 22:51:20.080827 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:20.080835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:20.081375 (+ 540us) server_negotiation.cc:407] Connection header received
0927 22:51:20.081504 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:20.081513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:20.081623 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:20.081747 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:20.082996 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.084040 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.085302 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.085788 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.085916 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:20.087935 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:20.087944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:20.087952 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:20.088007 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:20.091670 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:20.092300 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:20.092308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:20.092315 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:20.092498 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:20.092815 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:20.092823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:20.092828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:20.093233 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:20.093396 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:20.093824 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 22:51:20.094045 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:51:20.094556 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:20.080601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35035 (local address 127.0.97.1:34651)
0927 22:51:20.081287 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:20.081302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:20.081400 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:20.081785 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:20.081791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:20.082334 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:20.082855 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:20.082865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.084170 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.084176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:20.085161 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:20.085171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.086009 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.086016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:20.086197 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:20.086921 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:20.086954 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:20.087748 (+ 794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:20.091824 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:20.091830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:20.091833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:20.092187 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:20.092608 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:20.092611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:20.092613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:20.092718 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:20.093354 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:20.093357 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:20.093540 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 22:51:20.093748 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 22:51:20.094061 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":396,"threads_started":1}
I20260927 22:51:20.109905 2215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38493
I20260927 22:51:20.109987 2349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38493 every 8 connection(s)
I20260927 22:51:20.111097 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:20.112969 2350 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35035
I20260927 22:51:20.113149 2350 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:20.113758 2350 heartbeater.cc:511] Master 127.0.97.62:35035 requested a full tablet report, sending...
I20260927 22:51:20.115458 2144 ts_manager.cc:194] Registered new tserver with Master: 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1:38493)
I20260927 22:51:20.116794 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54183
I20260927 22:51:20.118113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 2215
I20260927 22:51:20.118309 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:20.118920 388 external_mini_cluster.cc:1428] Setting key efaaa55e13278e4ea4f28e982060c951
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:20.154935 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35035
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:20.461006 2354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:20.461253 2354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:20.461411 2354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:20.468358 2354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:20.468478 2354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:20.468629 2354 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:51:20.468707 2354 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:51:20.468837 2354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:20.478047 2354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:20.479223 2354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:20.480597 2354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:20.493378 2360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:20.493497 2359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:20.493423 2362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:20.494135 2354 server_base.cc:1034] running on GCE node
I20260927 22:51:20.494680 2354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:20.495633 2354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:20.496897 2354 hybrid_clock.cc:648] HybridClock initialized: now 1790549480496852 us; error 32 us; skew 500 ppm
I20260927 22:51:20.497223 2354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:20 dist-test-slave-df53 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:20.500339 2354 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:20.501726 2354 webserver.cc:460] Webserver started at http://127.0.97.2:34719/ using document root <none> and password file <none>
I20260927 22:51:20.502317 2354 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:20.502416 2354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:20.502660 2354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:20.505314 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "a2a239314ff74bcb8875449c1e1d170c"
format_stamp: "Formatted at 2026-09-27 22:51:20 on dist-test-slave-df53"
server_key: "e10b84efd461b37d813edf3c3822dc75"
server_key_iv: "43a6f130b17c38b764ca91541fe14a1d"
server_key_version: "encryptionkey@0"
I20260927 22:51:20.506088 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a2a239314ff74bcb8875449c1e1d170c"
format_stamp: "Formatted at 2026-09-27 22:51:20 on dist-test-slave-df53"
server_key: "e10b84efd461b37d813edf3c3822dc75"
server_key_iv: "43a6f130b17c38b764ca91541fe14a1d"
server_key_version: "encryptionkey@0"
I20260927 22:51:20.510785 2354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:51:20.513970 2369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:20.515264 2354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:51:20.515480 2354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "a2a239314ff74bcb8875449c1e1d170c"
format_stamp: "Formatted at 2026-09-27 22:51:20 on dist-test-slave-df53"
server_key: "e10b84efd461b37d813edf3c3822dc75"
server_key_iv: "43a6f130b17c38b764ca91541fe14a1d"
server_key_version: "encryptionkey@0"
I20260927 22:51:20.515679 2354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:20.530742 2354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:20.535171 2354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:20.535446 2354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:20.538128 2354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:20.538254 2354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:20.538408 2354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:20.538514 2354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:51:20 dist-test-slave-df53 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:20.564658 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:20.539319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35035 (local address 127.0.97.2:46115)
0927 22:51:20.540329 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:20.540387 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:20.543227 (+ 2840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:20.544058 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:20.544072 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:20.544825 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:20.545904 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:20.545939 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.547290 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.547294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:20.549621 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:20.549638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.550690 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.550705 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:20.551027 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:20.551838 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:20.551862 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:20.555524 (+ 3662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:20.559164 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:20.559180 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:20.559197 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:20.560033 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:20.560485 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:20.560493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:20.560500 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:20.560839 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:20.561705 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:20.561717 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:20.562124 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 22:51:20.562524 (+ 400us) client_negotiation.cc:240] Negotiation successful
0927 22:51:20.563052 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":433,"threads_started":1}
I20260927 22:51:20.565490 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:20.538816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46115 (local address 127.0.97.62:35035)
0927 22:51:20.539080 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:20.539089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:20.540523 (+ 1434us) server_negotiation.cc:407] Connection header received
0927 22:51:20.543551 (+ 3028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:20.543557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:20.543643 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:20.543759 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:20.546121 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.547115 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.549773 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.550180 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.550313 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:20.555802 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:20.555807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:20.555810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:20.555840 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:20.559015 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:20.560182 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:20.560190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:20.560193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:20.560352 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:20.561035 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:20.561042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:20.561048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:20.561466 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:20.561654 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:20.564844 (+ 3190us) server_negotiation.cc:299] Negotiation successful
0927 22:51:20.565113 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 22:51:20.588825 2354 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45985
I20260927 22:51:20.588843 2488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45985 every 8 connection(s)
I20260927 22:51:20.590436 2354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:20.591374 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 2354
I20260927 22:51:20.591557 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:20.591957 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:20.576548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35035 (local address 127.0.97.2:37219)
0927 22:51:20.577333 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:20.577355 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:20.577517 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:20.578087 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:20.578091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:20.578744 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:20.579059 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:20.579069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.580491 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.580498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:20.581715 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:20.581727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.582915 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.582921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:20.583070 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:20.583876 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:20.583892 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:20.585014 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:20.589178 (+ 4164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:20.589182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:20.589184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:20.589563 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:20.589981 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:20.589985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:20.589987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:20.590097 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:20.590755 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:20.590758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:20.590981 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 22:51:20.591165 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 22:51:20.591488 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":387,"threads_started":1}
I20260927 22:51:20.592221 388 external_mini_cluster.cc:1428] Setting key cb21aec5fe4b9957ab14f5161208f65f
I20260927 22:51:20.592543 2489 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35035
I20260927 22:51:20.592752 2489 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:20.592898 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:20.576624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37219 (local address 127.0.97.62:35035)
0927 22:51:20.576851 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:20.576860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:20.577446 (+ 586us) server_negotiation.cc:407] Connection header received
0927 22:51:20.577737 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:20.577745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:20.577832 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:20.577937 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:20.579203 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.580361 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.581862 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:20.582496 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:20.582643 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:20.585252 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:20.585262 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:20.585270 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:20.585307 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:20.589061 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:20.589713 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:20.589718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:20.589721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:20.589855 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:20.590212 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:20.590217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:20.590220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:20.590646 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:20.590789 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:20.592327 (+ 1538us) server_negotiation.cc:299] Negotiation successful
0927 22:51:20.592555 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:51:20.593537 2489 heartbeater.cc:511] Master 127.0.97.62:35035 requested a full tablet report, sending...
I20260927 22:51:20.596922 2143 ts_manager.cc:194] Registered new tserver with Master: a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985)
I20260927 22:51:20.597935 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46115
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:20.632915 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35035
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:20.943603 2493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:20.943915 2493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:20.944101 2493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:20.951179 2493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:20.951324 2493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:20.951488 2493 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:51:20.951574 2493 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:51:20.951689 2493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:20.961033 2493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:20.962327 2493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:20.963610 2493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:20.976158 2498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:20.976187 2499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:20.976541 2501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:20.976899 2493 server_base.cc:1034] running on GCE node
I20260927 22:51:20.977553 2493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:20.978482 2493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:20.979707 2493 hybrid_clock.cc:648] HybridClock initialized: now 1790549480979650 us; error 54 us; skew 500 ppm
I20260927 22:51:20.980037 2493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:20 dist-test-slave-df53 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:20.983107 2493 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:20.984546 2493 webserver.cc:460] Webserver started at http://127.0.97.3:37855/ using document root <none> and password file <none>
I20260927 22:51:20.985199 2493 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:20.985348 2493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:20.985725 2493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:20.988426 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "bd12d0edaba44dfeaa978c3c60c60445"
format_stamp: "Formatted at 2026-09-27 22:51:20 on dist-test-slave-df53"
server_key: "13123a5532bc456915a2f94a9962da56"
server_key_iv: "d85642c7630a1285d3c172e25aa96b0c"
server_key_version: "encryptionkey@0"
I20260927 22:51:20.989195 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bd12d0edaba44dfeaa978c3c60c60445"
format_stamp: "Formatted at 2026-09-27 22:51:20 on dist-test-slave-df53"
server_key: "13123a5532bc456915a2f94a9962da56"
server_key_iv: "d85642c7630a1285d3c172e25aa96b0c"
server_key_version: "encryptionkey@0"
I20260927 22:51:20.994047 2493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260927 22:51:20.997399 2508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:20.998770 2493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:51:20.998893 2493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "bd12d0edaba44dfeaa978c3c60c60445"
format_stamp: "Formatted at 2026-09-27 22:51:20 on dist-test-slave-df53"
server_key: "13123a5532bc456915a2f94a9962da56"
server_key_iv: "d85642c7630a1285d3c172e25aa96b0c"
server_key_version: "encryptionkey@0"
I20260927 22:51:20.999046 2493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:21.021765 2493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:21.025462 2493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:21.025796 2493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:21.028595 2493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:21.028684 2493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:21.028838 2493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:21.028951 2493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:51:21 dist-test-slave-df53 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:21.052835 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.029052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35173 (local address 127.0.97.62:35035)
0927 22:51:21.029258 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.029266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.030762 (+ 1496us) server_negotiation.cc:407] Connection header received
0927 22:51:21.032987 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.032993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.033073 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.033155 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:21.035700 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.036666 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.039086 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.039667 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.039806 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.045966 (+ 6160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:21.045976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:21.045983 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:21.046023 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:21.049166 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:21.050263 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:21.050267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:21.050270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:21.050406 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:21.051007 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:21.051011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:21.051013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:21.051306 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:21.051433 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.052446 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.052611 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:51:21.055003 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.029740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35035 (local address 127.0.97.3:35173)
0927 22:51:21.030577 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.030638 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.032623 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.033750 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.033758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.034637 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:21.035524 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.035546 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.036817 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.036827 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.038942 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.038953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.039909 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.039915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.040217 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.041261 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:21.041296 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:21.045630 (+ 4334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:21.049315 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:21.049329 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:21.049351 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:21.050110 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:21.050521 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:21.050529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:21.050535 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:21.050847 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:21.051452 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:21.051464 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:21.051992 (+ 528us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.052355 (+ 363us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.053019 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":372,"threads_started":1}
I20260927 22:51:21.078274 2493 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39163
I20260927 22:51:21.078377 2627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39163 every 8 connection(s)
I20260927 22:51:21.079063 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.062247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35035 (local address 127.0.97.3:34799)
0927 22:51:21.063060 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.063091 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.063263 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.063750 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.063754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.064965 (+ 1211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:21.065391 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.065408 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.067088 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.067093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.069130 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.069150 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.069954 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.069961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.070197 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.070903 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:21.070933 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:21.072344 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:21.076168 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:21.076173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:21.076181 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:21.076666 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:21.077100 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:21.077105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:21.077109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:21.077286 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:21.077941 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:21.077946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:21.078207 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.078472 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.078741 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":449,"threads_started":1}
I20260927 22:51:21.079710 2493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:21.079814 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.062114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34799 (local address 127.0.97.62:35035)
0927 22:51:21.062304 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.062311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.063207 (+ 896us) server_negotiation.cc:407] Connection header received
0927 22:51:21.063495 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.063502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.063598 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.063726 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:21.065573 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.066865 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.069270 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.069773 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.069914 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.072623 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:21.072632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:21.072638 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:21.072677 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:21.076029 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:21.076770 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:21.076775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:21.076778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:21.076930 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:21.077463 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:21.077470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:21.077476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:21.077815 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:21.077967 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.079377 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.079524 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:51:21.081928 2628 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35035
I20260927 22:51:21.082165 2628 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:21.082794 2628 heartbeater.cc:511] Master 127.0.97.62:35035 requested a full tablet report, sending...
I20260927 22:51:21.084020 2143 ts_manager.cc:194] Registered new tserver with Master: bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163)
I20260927 22:51:21.084771 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35173
I20260927 22:51:21.088533 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 2493
I20260927 22:51:21.088704 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:21.089125 388 external_mini_cluster.cc:1428] Setting key 3938107f18966f433f88d360b348f07c
I20260927 22:51:21.091818 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:51:21 dist-test-slave-df53 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260927 22:51:21.119674 2350 heartbeater.cc:503] Master 127.0.97.62:35035 was elected leader, sending a full tablet report...
Sep 27 22:51:21 dist-test-slave-df53 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:21.137279 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.124057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.0.97.62:35035)
0927 22:51:21.124277 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.124284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.124601 (+ 317us) server_negotiation.cc:407] Connection header received
0927 22:51:21.124734 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.124738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.124796 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.124893 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:21.126217 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.126914 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.128057 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.128464 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.128551 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.131998 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:21.132003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:21.132006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:21.132038 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:21.134837 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:21.135431 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:21.135439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:21.135445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:21.135582 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:21.135957 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:21.135962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:21.135966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:21.136215 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:21.136459 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.136747 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.136960 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:21.140852 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:21.147365 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:21.170485 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.158836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38493 (local address 127.0.0.1:57370)
0927 22:51:21.159851 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.159901 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.160104 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.161013 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.161018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.161045 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:21.161487 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.161502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.163622 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.163629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.166621 (+ 2992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.166695 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.169357 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.169367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.169641 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.169716 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.169884 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.170098 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":1961,"threads_started":1}
I20260927 22:51:21.171334 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.158883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.0.97.1:38493)
0927 22:51:21.159510 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.159521 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.160293 (+ 772us) server_negotiation.cc:407] Connection header received
0927 22:51:21.160403 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.160408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.160580 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.160786 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:21.161711 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.163462 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.166862 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.168758 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.168918 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.169193 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.170865 (+ 1672us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.171039 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":330,"threads_started":1}
I20260927 22:51:21.172242 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.158935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39163 (local address 127.0.0.1:33676)
0927 22:51:21.161054 (+ 2119us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.161072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.161182 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.162285 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.162293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.162325 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:21.162810 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.162824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.165331 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.165339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.168319 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.168339 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.171172 (+ 2833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.171181 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.171456 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.171509 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.171621 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.171800 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1983,"thread_start_us":592,"threads_started":1}
I20260927 22:51:21.173365 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.159607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.0.97.3:39163)
0927 22:51:21.160304 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.160311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.161315 (+ 1004us) server_negotiation.cc:407] Connection header received
0927 22:51:21.161452 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.161462 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.161742 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.161959 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:21.163062 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.165114 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.168558 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.170559 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.170730 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.171007 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.172723 (+ 1716us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.172940 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":358,"threads_started":1}
I20260927 22:51:21.177105 2558 tablet_service.cc:1467] Processing CreateTablet for tablet c4f0902df5dd4a43b2d485b44027bf97 (DEFAULT_TABLE table=test-table [id=407d892aadc94b189ea77f04b3774cec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:21.178171 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.161170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45985 (local address 127.0.0.1:51736)
0927 22:51:21.171724 (+ 10554us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.171741 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.171824 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.173104 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.173108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.173134 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:21.173299 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.173313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.174646 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.174650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.175996 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.176009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.177481 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.177487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.177694 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.177726 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.177834 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.177999 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10461,"thread_start_us":12245,"threads_started":1}
I20260927 22:51:21.177681 2280 tablet_service.cc:1467] Processing CreateTablet for tablet c4f0902df5dd4a43b2d485b44027bf97 (DEFAULT_TABLE table=test-table [id=407d892aadc94b189ea77f04b3774cec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:21.178656 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f0902df5dd4a43b2d485b44027bf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:21.178874 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.161449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.0.97.2:45985)
0927 22:51:21.172578 (+ 11129us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.172588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.172617 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:51:21.172715 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.172723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.172931 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.173135 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:21.173478 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.174504 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.176180 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.177319 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.177468 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.177676 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.178460 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.178619 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10966,"thread_start_us":10809,"threads_started":1}
I20260927 22:51:21.178964 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f0902df5dd4a43b2d485b44027bf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:21.181707 2419 tablet_service.cc:1467] Processing CreateTablet for tablet c4f0902df5dd4a43b2d485b44027bf97 (DEFAULT_TABLE table=test-table [id=407d892aadc94b189ea77f04b3774cec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:21.182853 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f0902df5dd4a43b2d485b44027bf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:21.192600 2647 tablet_bootstrap.cc:492] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: Bootstrap starting.
I20260927 22:51:21.195621 2647 tablet_bootstrap.cc:654] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:21.196555 2647 log.cc:824] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:21.197999 2648 tablet_bootstrap.cc:492] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: Bootstrap starting.
I20260927 22:51:21.200381 2647 tablet_bootstrap.cc:492] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: No bootstrap required, opened a new log
I20260927 22:51:21.200650 2647 ts_tablet_manager.cc:1397] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:51:21.201310 2648 tablet_bootstrap.cc:654] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:21.202286 2649 tablet_bootstrap.cc:492] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: Bootstrap starting.
I20260927 22:51:21.202559 2648 log.cc:824] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:21.205745 2648 tablet_bootstrap.cc:492] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: No bootstrap required, opened a new log
I20260927 22:51:21.205834 2649 tablet_bootstrap.cc:654] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:21.206005 2648 ts_tablet_manager.cc:1397] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260927 22:51:21.206986 2649 log.cc:824] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:21.209259 2647 raft_consensus.cc:348] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } }
I20260927 22:51:21.209591 2647 raft_consensus.cc:374] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:21.209734 2647 raft_consensus.cc:729] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a239314ff74bcb8875449c1e1d170c, State: Initialized, Role: FOLLOWER
I20260927 22:51:21.210319 2647 consensus_queue.cc:260] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } }
I20260927 22:51:21.210911 2649 tablet_bootstrap.cc:492] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: No bootstrap required, opened a new log
I20260927 22:51:21.211210 2649 ts_tablet_manager.cc:1397] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260927 22:51:21.211277 2489 heartbeater.cc:503] Master 127.0.97.62:35035 was elected leader, sending a full tablet report...
I20260927 22:51:21.211939 2647 ts_tablet_manager.cc:1428] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 22:51:21.216434 2648 raft_consensus.cc:348] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } }
I20260927 22:51:21.216825 2648 raft_consensus.cc:374] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:21.216929 2648 raft_consensus.cc:729] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f76d17dbf9945388d5d9d18f5bf1dea, State: Initialized, Role: FOLLOWER
I20260927 22:51:21.217509 2648 consensus_queue.cc:260] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } }
I20260927 22:51:21.219008 2648 ts_tablet_manager.cc:1428] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260927 22:51:21.220994 2649 raft_consensus.cc:348] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } }
I20260927 22:51:21.221379 2649 raft_consensus.cc:374] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:21.221483 2649 raft_consensus.cc:729] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd12d0edaba44dfeaa978c3c60c60445, State: Initialized, Role: FOLLOWER
I20260927 22:51:21.222069 2649 consensus_queue.cc:260] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } }
I20260927 22:51:21.223003 2628 heartbeater.cc:503] Master 127.0.97.62:35035 was elected leader, sending a full tablet report...
I20260927 22:51:21.223351 2649 ts_tablet_manager.cc:1428] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260927 22:51:21.237802 2653 raft_consensus.cc:482] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:21.238049 2653 raft_consensus.cc:504] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } }
I20260927 22:51:21.239461 2653 leader_election.cc:288] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1:38493), bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163)
I20260927 22:51:21.248643 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.239823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45319 (local address 127.0.97.3:39163)
0927 22:51:21.240006 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.240016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.240627 (+ 611us) server_negotiation.cc:407] Connection header received
0927 22:51:21.240758 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.240764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.240835 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.240934 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:21.241895 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.242975 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.246224 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.247430 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.247558 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.247656 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.248266 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.248420 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:51:21.248643 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.239914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35513 (local address 127.0.97.1:38493)
0927 22:51:21.240112 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.240121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.240570 (+ 449us) server_negotiation.cc:407] Connection header received
0927 22:51:21.240758 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.240764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.240851 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.240934 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:21.241909 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.242980 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.246230 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.247428 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.247558 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.247654 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.248266 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.248392 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 22:51:21.248963 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.239709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38493 (local address 127.0.97.2:35513)
0927 22:51:21.240461 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.240481 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.240628 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.240970 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.240975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.241012 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:21.241758 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.241771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.243214 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.243220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.246064 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.246077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.247779 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.247783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.248032 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.248129 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.248229 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.248475 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":502,"spinlock_wait_cycles":896,"thread_start_us":426,"threads_started":1}
I20260927 22:51:21.248987 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.239781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39163 (local address 127.0.97.2:45319)
0927 22:51:21.240524 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.240541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.240639 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.240970 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.240975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.241018 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:21.241693 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.241712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.243216 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.243220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.246063 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.246077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.247758 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.247770 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.248026 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.248129 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.248229 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.248475 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":417,"spinlock_wait_cycles":6272,"thread_start_us":430,"threads_started":1}
I20260927 22:51:21.249966 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f0902df5dd4a43b2d485b44027bf97" candidate_uuid: "a2a239314ff74bcb8875449c1e1d170c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" is_pre_election: true
I20260927 22:51:21.249967 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f0902df5dd4a43b2d485b44027bf97" candidate_uuid: "a2a239314ff74bcb8875449c1e1d170c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd12d0edaba44dfeaa978c3c60c60445" is_pre_election: true
I20260927 22:51:21.250403 2578 raft_consensus.cc:2454] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a239314ff74bcb8875449c1e1d170c in term 0.
I20260927 22:51:21.250406 2305 raft_consensus.cc:2454] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a239314ff74bcb8875449c1e1d170c in term 0.
I20260927 22:51:21.250985 2373 leader_election.cc:302] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a239314ff74bcb8875449c1e1d170c, bd12d0edaba44dfeaa978c3c60c60445; no voters:
I20260927 22:51:21.251361 2653 raft_consensus.cc:2781] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:21.251506 2653 raft_consensus.cc:482] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:21.251608 2653 raft_consensus.cc:3037] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:21.253695 2653 raft_consensus.cc:504] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } }
I20260927 22:51:21.254215 2653 leader_election.cc:288] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [CANDIDATE]: Term 1 election: Requested vote from peers 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1:38493), bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163)
I20260927 22:51:21.254606 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f0902df5dd4a43b2d485b44027bf97" candidate_uuid: "a2a239314ff74bcb8875449c1e1d170c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea"
I20260927 22:51:21.254635 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f0902df5dd4a43b2d485b44027bf97" candidate_uuid: "a2a239314ff74bcb8875449c1e1d170c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd12d0edaba44dfeaa978c3c60c60445"
I20260927 22:51:21.254799 2305 raft_consensus.cc:3037] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:21.254799 2578 raft_consensus.cc:3037] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:21.257928 2578 raft_consensus.cc:2454] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a239314ff74bcb8875449c1e1d170c in term 1.
I20260927 22:51:21.257915 2305 raft_consensus.cc:2454] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a239314ff74bcb8875449c1e1d170c in term 1.
I20260927 22:51:21.258277 2370 leader_election.cc:302] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f76d17dbf9945388d5d9d18f5bf1dea, a2a239314ff74bcb8875449c1e1d170c; no voters:
I20260927 22:51:21.258548 2653 raft_consensus.cc:2781] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:21.258950 2653 raft_consensus.cc:686] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 1 LEADER]: Becoming Leader. State: Replica: a2a239314ff74bcb8875449c1e1d170c, State: Running, Role: LEADER
I20260927 22:51:21.259364 2653 consensus_queue.cc:237] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } }
I20260927 22:51:21.263777 2143 catalog_manager.cc:5495] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a239314ff74bcb8875449c1e1d170c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a2a239314ff74bcb8875449c1e1d170c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } health_report { overall_health: HEALTHY } } }
Sep 27 22:51:21 dist-test-slave-df53 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:21.321551 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.308762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35035 (local address 127.0.0.1:51814)
0927 22:51:21.309480 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.309496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.309675 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.309987 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.309991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.310612 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:21.310893 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.310902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.311855 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.311859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.312842 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.312852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.313529 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.313534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.313800 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.314496 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:21.314513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:21.316550 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:21.319207 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:21.319216 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:21.319222 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:21.319692 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:21.320027 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:21.320030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:21.320032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:21.320171 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:21.320678 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:21.320684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:21.320812 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.320956 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.321189 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":450,"threads_started":1}
I20260927 22:51:21.321785 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.308763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.0.97.62:35035)
0927 22:51:21.308985 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.308994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.309628 (+ 634us) server_negotiation.cc:407] Connection header received
0927 22:51:21.309798 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.309802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.309859 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.309948 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:21.311013 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.311711 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.312957 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.313329 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.313422 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.316727 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:21.316732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:21.316735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:21.316762 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:21.319061 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:21.319791 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:21.319794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:21.319797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:21.319906 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:21.320265 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:21.320268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:21.320270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:21.320567 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:21.320677 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.321048 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.321338 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":87}
I20260927 22:51:21.326493 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51814:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:51:21.327469 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260927 22:51:21.331471 2629 tablet.cc:2369] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:21.333381 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 10256d1ee8594b8a8f569b3b7f2b525e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d5f9d3573a24e93876046216f75d525]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:51:21.333495 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 10256d1ee8594b8a8f569b3b7f2b525e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d5f9d3573a24e93876046216f75d525]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:51:21.334455 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10256d1ee8594b8a8f569b3b7f2b525e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:21.334635 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10256d1ee8594b8a8f569b3b7f2b525e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:21.334363 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 10256d1ee8594b8a8f569b3b7f2b525e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d5f9d3573a24e93876046216f75d525]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:51:21.335296 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10256d1ee8594b8a8f569b3b7f2b525e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:21.340098 2648 tablet_bootstrap.cc:492] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: Bootstrap starting.
I20260927 22:51:21.341349 2647 tablet_bootstrap.cc:492] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: Bootstrap starting.
I20260927 22:51:21.341393 2648 tablet_bootstrap.cc:654] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:21.341943 2649 tablet_bootstrap.cc:492] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: Bootstrap starting.
I20260927 22:51:21.342991 2647 tablet_bootstrap.cc:654] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:21.343500 2649 tablet_bootstrap.cc:654] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:21.343712 2648 tablet_bootstrap.cc:492] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: No bootstrap required, opened a new log
I20260927 22:51:21.343886 2648 ts_tablet_manager.cc:1397] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:51:21.344605 2648 raft_consensus.cc:348] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.344882 2648 raft_consensus.cc:374] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:21.344988 2648 raft_consensus.cc:729] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f76d17dbf9945388d5d9d18f5bf1dea, State: Initialized, Role: FOLLOWER
I20260927 22:51:21.345324 2647 tablet_bootstrap.cc:492] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: No bootstrap required, opened a new log
I20260927 22:51:21.345302 2648 consensus_queue.cc:260] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.345472 2647 ts_tablet_manager.cc:1397] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:51:21.345693 2649 tablet_bootstrap.cc:492] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: No bootstrap required, opened a new log
I20260927 22:51:21.345818 2649 ts_tablet_manager.cc:1397] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260927 22:51:21.345849 2654 tablet_replica.cc:439] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } }
I20260927 22:51:21.346204 2654 tablet_replica.cc:442] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 22:51:21.346436 2490 tablet.cc:2369] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:21.346598 2649 raft_consensus.cc:348] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.346868 2649 raft_consensus.cc:374] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:21.346974 2649 raft_consensus.cc:729] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd12d0edaba44dfeaa978c3c60c60445, State: Initialized, Role: FOLLOWER
I20260927 22:51:21.347188 2649 consensus_queue.cc:260] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.347400 2647 raft_consensus.cc:348] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.347637 2647 raft_consensus.cc:374] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:21.347729 2647 raft_consensus.cc:729] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a239314ff74bcb8875449c1e1d170c, State: Initialized, Role: FOLLOWER
I20260927 22:51:21.347592 2655 tablet_replica.cc:439] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } }
I20260927 22:51:21.347903 2655 tablet_replica.cc:442] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:21.347963 2648 ts_tablet_manager.cc:1428] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260927 22:51:21.347941 2647 consensus_queue.cc:260] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.348356 2653 tablet_replica.cc:439] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } }
I20260927 22:51:21.348506 2647 ts_tablet_manager.cc:1428] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260927 22:51:21.348613 2653 tablet_replica.cc:442] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:21.348763 2649 ts_tablet_manager.cc:1428] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260927 22:51:21.363300 2351 tablet.cc:2369] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:21.430738 2654 raft_consensus.cc:482] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:21.431020 2654 raft_consensus.cc:504] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.432607 2654 leader_election.cc:288] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985), bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163)
I20260927 22:51:21.440389 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.433092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48473 (local address 127.0.97.3:39163)
0927 22:51:21.433261 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.433269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.434142 (+ 873us) server_negotiation.cc:407] Connection header received
0927 22:51:21.434299 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.434325 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.434426 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.434535 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:21.435247 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.436263 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.438404 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.439392 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.439491 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.439558 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.439972 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.440114 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:51:21.440465 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.433186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39163 (local address 127.0.97.1:48473)
0927 22:51:21.434046 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.434060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.434172 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.434545 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.434551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.434581 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:21.435069 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.435087 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.436407 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.436410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.438267 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.438278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.439600 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.439609 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.439789 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.439838 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.439909 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.439994 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":1,"thread_start_us":526,"threads_started":1}
I20260927 22:51:21.440470 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.433055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44891 (local address 127.0.97.2:45985)
0927 22:51:21.433258 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.433266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.433798 (+ 532us) server_negotiation.cc:407] Connection header received
0927 22:51:21.434401 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.434404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.434459 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.434562 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:21.435295 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.436239 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.438758 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.439550 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.439686 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.439785 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.440083 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.440224 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:51:21.440644 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.432917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45985 (local address 127.0.97.1:44891)
0927 22:51:21.433691 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.433710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.434253 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.434607 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.434611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.434638 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:21.435116 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.435129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.436369 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.436374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.438615 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.438628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.439758 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.439762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.439878 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.439952 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.440055 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.440173 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":1,"spinlock_wait_cycles":24448,"thread_start_us":412,"threads_started":1}
I20260927 22:51:21.441457 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10256d1ee8594b8a8f569b3b7f2b525e" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd12d0edaba44dfeaa978c3c60c60445" is_pre_election: true
I20260927 22:51:21.441568 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10256d1ee8594b8a8f569b3b7f2b525e" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a239314ff74bcb8875449c1e1d170c" is_pre_election: true
I20260927 22:51:21.441771 2578 raft_consensus.cc:2454] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 0.
I20260927 22:51:21.441998 2439 raft_consensus.cc:2454] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 0.
I20260927 22:51:21.442226 2234 leader_election.cc:302] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f76d17dbf9945388d5d9d18f5bf1dea, bd12d0edaba44dfeaa978c3c60c60445; no voters:
I20260927 22:51:21.442654 2654 raft_consensus.cc:2781] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:21.442798 2654 raft_consensus.cc:482] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:21.442911 2654 raft_consensus.cc:3037] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:21.449386 2654 raft_consensus.cc:504] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.450065 2654 leader_election.cc:288] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 election: Requested vote from peers a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985), bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163)
I20260927 22:51:21.450536 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10256d1ee8594b8a8f569b3b7f2b525e" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd12d0edaba44dfeaa978c3c60c60445"
I20260927 22:51:21.450564 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10256d1ee8594b8a8f569b3b7f2b525e" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a239314ff74bcb8875449c1e1d170c"
I20260927 22:51:21.450752 2578 raft_consensus.cc:3037] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:21.450788 2439 raft_consensus.cc:3037] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:21.452824 2578 raft_consensus.cc:2454] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 1.
I20260927 22:51:21.453249 2234 leader_election.cc:302] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f76d17dbf9945388d5d9d18f5bf1dea, bd12d0edaba44dfeaa978c3c60c60445; no voters:
I20260927 22:51:21.453534 2654 raft_consensus.cc:2781] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:21.454144 2654 raft_consensus.cc:686] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 LEADER]: Becoming Leader. State: Replica: 4f76d17dbf9945388d5d9d18f5bf1dea, State: Running, Role: LEADER
I20260927 22:51:21.454722 2654 consensus_queue.cc:237] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.458593 2671 tablet_replica.cc:439] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: TxnStatusTablet state changed. Reason: New leader 4f76d17dbf9945388d5d9d18f5bf1dea. Latest consensus state: current_term: 1 leader_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } }
I20260927 22:51:21.458981 2671 tablet_replica.cc:442] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:51:21.459669 2673 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:51:21.459915 2673 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:51:21.460341 2144 catalog_manager.cc:5495] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } health_report { overall_health: UNKNOWN } } }
I20260927 22:51:21.506845 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.500306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.97.1:38493)
0927 22:51:21.500465 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.500470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.500491 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:51:21.500799 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.500803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.500911 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.501035 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:21.501429 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.502393 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.504300 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.504858 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.504961 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.505539 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:21.506269 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:21.506393 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.506550 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.506611 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:51:21.506966 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.500139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38493 (local address 127.0.0.1:57382)
0927 22:51:21.500423 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.500445 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.500597 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.501086 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.501091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.501104 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:51:21.501290 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.501300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.502537 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.502544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.504147 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.504163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.505015 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.505022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.505242 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.505325 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:51:21.506391 (+ 1066us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:51:21.506411 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.506501 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.506649 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 22:51:21.508064 2439 raft_consensus.cc:2454] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 1.
I20260927 22:51:21.514842 2578 raft_consensus.cc:1261] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Refusing update from remote peer 4f76d17dbf9945388d5d9d18f5bf1dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:21.514860 2439 raft_consensus.cc:1261] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Refusing update from remote peer 4f76d17dbf9945388d5d9d18f5bf1dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:21.515789 2671 consensus_queue.cc:1035] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:21.516170 2654 consensus_queue.cc:1035] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:21.534273 2676 mvcc.cc:205] Tried to move back new op lower bound from 7334090676276129792 to 7334090676042444800. Current Snapshot: MvccSnapshot[applied={T|T < 7334090676276129792}]
I20260927 22:51:21.535439 2653 tablet_replica.cc:439] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: TxnStatusTablet state changed. Reason: New leader 4f76d17dbf9945388d5d9d18f5bf1dea. Latest consensus state: current_term: 1 leader_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } }
I20260927 22:51:21.535729 2653 tablet_replica.cc:442] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:21.537901 2654 tablet_replica.cc:439] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } }
I20260927 22:51:21.538193 2654 tablet_replica.cc:442] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:51:21.546627 2678 mvcc.cc:205] Tried to move back new op lower bound from 7334090676276129792 to 7334090676042444800. Current Snapshot: MvccSnapshot[applied={T|T < 7334090676276129792}]
I20260927 22:51:21.553105 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.544264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51744 (local address 127.0.97.2:45985)
0927 22:51:21.544430 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.544436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.544704 (+ 268us) server_negotiation.cc:407] Connection header received
0927 22:51:21.544941 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.544946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.545041 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.545175 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:21.545749 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.547014 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.550010 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.550644 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.550758 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.551376 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:21.552034 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:21.552163 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.552741 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.552815 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:51:21.559002 2653 tablet_replica.cc:439] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } }
I20260927 22:51:21.559315 2653 tablet_replica.cc:442] T 10256d1ee8594b8a8f569b3b7f2b525e P a2a239314ff74bcb8875449c1e1d170c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:21.570945 2655 tablet_replica.cc:439] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: TxnStatusTablet state changed. Reason: New leader 4f76d17dbf9945388d5d9d18f5bf1dea. Latest consensus state: current_term: 1 leader_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } }
I20260927 22:51:21.571241 2655 tablet_replica.cc:442] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:21.575249 2671 tablet_replica.cc:439] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } }
I20260927 22:51:21.577889 2671 tablet_replica.cc:442] T 10256d1ee8594b8a8f569b3b7f2b525e P 4f76d17dbf9945388d5d9d18f5bf1dea: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:51:21.583490 2655 tablet_replica.cc:439] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } } }
I20260927 22:51:21.583736 2655 tablet_replica.cc:442] T 10256d1ee8594b8a8f569b3b7f2b525e P bd12d0edaba44dfeaa978c3c60c60445: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:51:21.593968 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.586485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56961 (local address 127.0.97.1:38493)
0927 22:51:21.586902 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.586908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.587486 (+ 578us) server_negotiation.cc:407] Connection header received
0927 22:51:21.587659 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.587662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.587740 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.587875 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:21.588511 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.589711 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.591829 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.592516 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.592649 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.593156 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:21.593281 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:21.593372 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.593568 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.593666 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260927 22:51:21.594252 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.586388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38493 (local address 127.0.97.2:56961)
0927 22:51:21.587379 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.587398 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.587541 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.587896 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.587901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.587914 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:51:21.588350 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.588363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.589858 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.589864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.591682 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.591696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.592705 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.592713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.592906 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.592969 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:51:21.593388 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:51:21.593399 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.593483 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.593643 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":670,"threads_started":1}
I20260927 22:51:21.603837 2578 raft_consensus.cc:1261] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Refusing update from remote peer a2a239314ff74bcb8875449c1e1d170c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:21.604401 2305 raft_consensus.cc:1261] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Refusing update from remote peer a2a239314ff74bcb8875449c1e1d170c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:21.604908 2658 consensus_queue.cc:1035] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:21.605865 2658 consensus_queue.cc:1035] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:21.626731 2677 mvcc.cc:205] Tried to move back new op lower bound from 7334090676643250176 to 7334090675242221568. Current Snapshot: MvccSnapshot[applied={T|T < 7334090676643250176}]
I20260927 22:51:21.657425 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.650165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33483 (local address 127.0.97.2:45985)
0927 22:51:21.650396 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.650402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.651679 (+ 1277us) server_negotiation.cc:407] Connection header received
0927 22:51:21.651919 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.651924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.652004 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.652104 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:21.652638 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.653740 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.655234 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.655927 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.656052 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.656625 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:21.656745 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:21.656840 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.657050 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.657129 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:21.658063 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.650424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45985 (local address 127.0.97.1:33483)
0927 22:51:21.651452 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.651581 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.651745 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.652125 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.652128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.652138 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:51:21.652480 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.652488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.653880 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.653884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.655104 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.655116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.656107 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.656114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.656347 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.656435 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:51:21.656856 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:51:21.656870 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.657601 (+ 731us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.657771 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":666,"threads_started":1}
I20260927 22:51:21.693338 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:21.693639 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:21.697471 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f: Sending DeleteTablet for 3 replicas of tablet c4f0902df5dd4a43b2d485b44027bf97
I20260927 22:51:21.698655 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f0902df5dd4a43b2d485b44027bf97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33676
I20260927 22:51:21.698907 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f0902df5dd4a43b2d485b44027bf97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51736
I20260927 22:51:21.701043 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:51:21.701107 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f0902df5dd4a43b2d485b44027bf97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57370
W20260927 22:51:21.702044 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:21.711925 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3e9f43def842d1a194b498953ac2c8 (DEFAULT_TABLE table=test-table [id=fd315cb194e3478984bf7e2c448902bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:21.712390 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3e9f43def842d1a194b498953ac2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:21.714681 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3e9f43def842d1a194b498953ac2c8 (DEFAULT_TABLE table=test-table [id=fd315cb194e3478984bf7e2c448902bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:21.715126 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3e9f43def842d1a194b498953ac2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:21.722368 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3e9f43def842d1a194b498953ac2c8 (DEFAULT_TABLE table=test-table [id=fd315cb194e3478984bf7e2c448902bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:21.722849 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3e9f43def842d1a194b498953ac2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:21.730031 2699 tablet_replica.cc:330] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: stopping tablet replica
I20260927 22:51:21.730782 2699 raft_consensus.cc:2229] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:21.733557 2699 raft_consensus.cc:2258] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:21.735859 2649 tablet_bootstrap.cc:492] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445: Bootstrap starting.
I20260927 22:51:21.737265 2699 ts_tablet_manager.cc:1905] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:21.737560 2649 tablet_bootstrap.cc:654] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:21.742229 2648 tablet_bootstrap.cc:492] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea: Bootstrap starting.
I20260927 22:51:21.742641 2647 tablet_bootstrap.cc:492] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c: Bootstrap starting.
I20260927 22:51:21.743803 2648 tablet_bootstrap.cc:654] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:21.744328 2647 tablet_bootstrap.cc:654] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:21.760533 2699 ts_tablet_manager.cc:1918] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:51:21.760775 2699 log.cc:1196] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/c4f0902df5dd4a43b2d485b44027bf97
I20260927 22:51:21.761421 2699 ts_tablet_manager.cc:1939] T c4f0902df5dd4a43b2d485b44027bf97 P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting consensus metadata
I20260927 22:51:21.763679 2128 catalog_manager.cc:4841] TS 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1:38493): tablet c4f0902df5dd4a43b2d485b44027bf97 (table test-table [id=407d892aadc94b189ea77f04b3774cec]) successfully deleted
I20260927 22:51:21.770845 2698 tablet_replica.cc:330] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: stopping tablet replica
I20260927 22:51:21.771445 2698 raft_consensus.cc:2229] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:21.771530 2649 tablet_bootstrap.cc:492] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445: No bootstrap required, opened a new log
I20260927 22:51:21.772143 2698 raft_consensus.cc:2258] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:21.772964 2647 tablet_bootstrap.cc:492] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c: No bootstrap required, opened a new log
I20260927 22:51:21.773213 2649 ts_tablet_manager.cc:1397] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.004s
I20260927 22:51:21.773437 2647 ts_tablet_manager.cc:1397] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.005s
I20260927 22:51:21.774104 2698 ts_tablet_manager.cc:1905] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:21.775247 2649 raft_consensus.cc:348] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.775578 2649 raft_consensus.cc:374] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:21.775704 2649 raft_consensus.cc:729] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd12d0edaba44dfeaa978c3c60c60445, State: Initialized, Role: FOLLOWER
I20260927 22:51:21.776075 2649 consensus_queue.cc:260] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.777113 2649 ts_tablet_manager.cc:1428] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:51:21.778461 2648 tablet_bootstrap.cc:492] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea: No bootstrap required, opened a new log
I20260927 22:51:21.778669 2648 ts_tablet_manager.cc:1397] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.007s
I20260927 22:51:21.779434 2648 raft_consensus.cc:348] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.779784 2648 raft_consensus.cc:374] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:21.779918 2648 raft_consensus.cc:729] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f76d17dbf9945388d5d9d18f5bf1dea, State: Initialized, Role: FOLLOWER
I20260927 22:51:21.780251 2648 consensus_queue.cc:260] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.782039 2697 tablet_replica.cc:330] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: stopping tablet replica
I20260927 22:51:21.783126 2648 ts_tablet_manager.cc:1428] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260927 22:51:21.774345 2647 raft_consensus.cc:348] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.785444 2647 raft_consensus.cc:374] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:21.785554 2647 raft_consensus.cc:729] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a239314ff74bcb8875449c1e1d170c, State: Initialized, Role: FOLLOWER
I20260927 22:51:21.785998 2697 raft_consensus.cc:2229] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:21.785955 2647 consensus_queue.cc:260] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.786360 2697 raft_consensus.cc:2258] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:21.788375 2697 ts_tablet_manager.cc:1905] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:21.789645 2698 ts_tablet_manager.cc:1918] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:51:21.789836 2698 log.cc:1196] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/c4f0902df5dd4a43b2d485b44027bf97
I20260927 22:51:21.790373 2698 ts_tablet_manager.cc:1939] T c4f0902df5dd4a43b2d485b44027bf97 P a2a239314ff74bcb8875449c1e1d170c: Deleting consensus metadata
I20260927 22:51:21.792047 2128 catalog_manager.cc:4841] TS a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985): tablet c4f0902df5dd4a43b2d485b44027bf97 (table test-table [id=407d892aadc94b189ea77f04b3774cec]) successfully deleted
I20260927 22:51:21.793748 2647 ts_tablet_manager.cc:1428] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20260927 22:51:21.795506 2697 ts_tablet_manager.cc:1918] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:51:21.795702 2697 log.cc:1196] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/c4f0902df5dd4a43b2d485b44027bf97
I20260927 22:51:21.796284 2697 ts_tablet_manager.cc:1939] T c4f0902df5dd4a43b2d485b44027bf97 P bd12d0edaba44dfeaa978c3c60c60445: Deleting consensus metadata
I20260927 22:51:21.797940 2131 catalog_manager.cc:4841] TS bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163): tablet c4f0902df5dd4a43b2d485b44027bf97 (table test-table [id=407d892aadc94b189ea77f04b3774cec]) successfully deleted
I20260927 22:51:21.820487 2654 raft_consensus.cc:482] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:21.820691 2654 raft_consensus.cc:504] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.821497 2654 leader_election.cc:288] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985), bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163)
I20260927 22:51:21.822027 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3e9f43def842d1a194b498953ac2c8" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a239314ff74bcb8875449c1e1d170c" is_pre_election: true
I20260927 22:51:21.822185 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3e9f43def842d1a194b498953ac2c8" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd12d0edaba44dfeaa978c3c60c60445" is_pre_election: true
I20260927 22:51:21.822264 2439 raft_consensus.cc:2454] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 0.
I20260927 22:51:21.822430 2578 raft_consensus.cc:2454] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 0.
I20260927 22:51:21.822904 2234 leader_election.cc:302] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f76d17dbf9945388d5d9d18f5bf1dea, bd12d0edaba44dfeaa978c3c60c60445; no voters:
I20260927 22:51:21.823220 2654 raft_consensus.cc:2781] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:21.823333 2654 raft_consensus.cc:482] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:21.823424 2654 raft_consensus.cc:3037] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:21.826674 2654 raft_consensus.cc:504] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.827780 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3e9f43def842d1a194b498953ac2c8" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd12d0edaba44dfeaa978c3c60c60445"
I20260927 22:51:21.827792 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3e9f43def842d1a194b498953ac2c8" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a239314ff74bcb8875449c1e1d170c"
I20260927 22:51:21.827975 2578 raft_consensus.cc:3037] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:21.827987 2439 raft_consensus.cc:3037] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:21.830852 2578 raft_consensus.cc:2454] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 1.
I20260927 22:51:21.831202 2439 raft_consensus.cc:2454] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 1.
I20260927 22:51:21.831283 2234 leader_election.cc:302] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f76d17dbf9945388d5d9d18f5bf1dea, bd12d0edaba44dfeaa978c3c60c60445; no voters:
I20260927 22:51:21.831588 2671 raft_consensus.cc:2781] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:21.831753 2671 raft_consensus.cc:686] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 LEADER]: Becoming Leader. State: Replica: 4f76d17dbf9945388d5d9d18f5bf1dea, State: Running, Role: LEADER
I20260927 22:51:21.832046 2671 consensus_queue.cc:237] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:21.832204 2654 leader_election.cc:288] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 election: Requested vote from peers a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985), bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163)
I20260927 22:51:21.843469 2144 catalog_manager.cc:5495] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } health_report { overall_health: UNKNOWN } } }
I20260927 22:51:21.890172 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.883732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.0.97.1:38493)
0927 22:51:21.883903 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.883909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.884034 (+ 125us) server_negotiation.cc:407] Connection header received
0927 22:51:21.884215 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.884220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.884305 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.884424 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:21.884765 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.886030 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.887980 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.888701 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.888849 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.889312 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:21.889464 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:21.889553 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.889759 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.889839 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:51:21.905757 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.899514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47959 (local address 127.0.97.1:38493)
0927 22:51:21.899668 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:21.899674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:21.899737 (+ 63us) server_negotiation.cc:407] Connection header received
0927 22:51:21.899943 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:21.899949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:21.900028 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:21.900135 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:21.900496 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.901655 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.903647 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.904331 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.904465 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.904909 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:21.905077 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:21.905182 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:21.905352 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 22:51:21.905428 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:51:21.905790 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:21.899422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38493 (local address 127.0.97.1:47959)
0927 22:51:21.899669 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:21.899687 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:21.899800 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:21.900161 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:21.900166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:21.900179 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:51:21.900354 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.900364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.901787 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.901793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.903486 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:21.903500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:21.904490 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:21.904500 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:21.904694 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:21.904743 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:51:21.905193 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:51:21.905203 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:51:21.905307 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:51:21.905450 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 22:51:21.915724 2439 raft_consensus.cc:1261] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Refusing update from remote peer 4f76d17dbf9945388d5d9d18f5bf1dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:21.915807 2578 raft_consensus.cc:1261] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Refusing update from remote peer 4f76d17dbf9945388d5d9d18f5bf1dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:21.916386 2654 consensus_queue.cc:1035] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:21.916872 2688 consensus_queue.cc:1035] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:51:21.954039 2676 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:51:23.562019 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:23.562218 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:23.566006 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f: Sending DeleteTablet for 3 replicas of tablet 5f3e9f43def842d1a194b498953ac2c8
I20260927 22:51:23.566715 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3e9f43def842d1a194b498953ac2c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57370
I20260927 22:51:23.566912 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3e9f43def842d1a194b498953ac2c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51736
I20260927 22:51:23.567160 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3e9f43def842d1a194b498953ac2c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33676
I20260927 22:51:23.579658 2728 tablet_replica.cc:330] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c: stopping tablet replica
I20260927 22:51:23.579998 2728 raft_consensus.cc:2229] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:23.580153 2729 tablet_replica.cc:330] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445: stopping tablet replica
I20260927 22:51:23.580310 2727 tablet_replica.cc:330] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea: stopping tablet replica
I20260927 22:51:23.580329 2728 raft_consensus.cc:2258] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:23.580431 2729 raft_consensus.cc:2229] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:23.580536 2727 raft_consensus.cc:2229] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:23.581068 2729 raft_consensus.cc:2258] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:23.581200 2727 raft_consensus.cc:2258] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:23.581377 2728 ts_tablet_manager.cc:1905] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:23.582350 2729 ts_tablet_manager.cc:1905] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:23.583253 2727 ts_tablet_manager.cc:1905] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:23.589876 2727 ts_tablet_manager.cc:1918] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:51:23.590071 2727 log.cc:1196] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/5f3e9f43def842d1a194b498953ac2c8
I20260927 22:51:23.590108 2729 ts_tablet_manager.cc:1918] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:51:23.590294 2729 log.cc:1196] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/5f3e9f43def842d1a194b498953ac2c8
I20260927 22:51:23.590553 2727 ts_tablet_manager.cc:1939] T 5f3e9f43def842d1a194b498953ac2c8 P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting consensus metadata
I20260927 22:51:23.590843 2729 ts_tablet_manager.cc:1939] T 5f3e9f43def842d1a194b498953ac2c8 P bd12d0edaba44dfeaa978c3c60c60445: Deleting consensus metadata
I20260927 22:51:23.591630 2128 catalog_manager.cc:4841] TS 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1:38493): tablet 5f3e9f43def842d1a194b498953ac2c8 (table test-table [id=fd315cb194e3478984bf7e2c448902bf]) successfully deleted
I20260927 22:51:23.591969 2131 catalog_manager.cc:4841] TS bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163): tablet 5f3e9f43def842d1a194b498953ac2c8 (table test-table [id=fd315cb194e3478984bf7e2c448902bf]) successfully deleted
I20260927 22:51:23.592267 2728 ts_tablet_manager.cc:1918] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:51:23.592391 2728 log.cc:1196] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/5f3e9f43def842d1a194b498953ac2c8
I20260927 22:51:23.592774 2728 ts_tablet_manager.cc:1939] T 5f3e9f43def842d1a194b498953ac2c8 P a2a239314ff74bcb8875449c1e1d170c: Deleting consensus metadata
I20260927 22:51:23.593714 2128 catalog_manager.cc:4841] TS a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985): tablet 5f3e9f43def842d1a194b498953ac2c8 (table test-table [id=fd315cb194e3478984bf7e2c448902bf]) successfully deleted
Sep 27 22:51:23 dist-test-slave-df53 krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549483, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:51:23 dist-test-slave-df53 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549483, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:23.630108 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.612591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.0.97.62:35035)
0927 22:51:23.613088 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:23.613092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:23.613439 (+ 347us) server_negotiation.cc:407] Connection header received
0927 22:51:23.613583 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:23.613620 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:23.613688 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:23.613796 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:23.614979 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.616321 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.617842 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.618215 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.618316 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.625465 (+ 7149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:23.625470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:23.625473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:23.625503 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:23.628078 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:23.628609 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:23.628613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:23.628615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:23.628720 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:23.629022 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:23.629025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:23.629026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:23.629253 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:23.629411 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:23.629695 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 22:51:23.629853 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":234,"threads_started":1}
I20260927 22:51:23.633406 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:23.633967 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:23.639818 2280 tablet_service.cc:1467] Processing CreateTablet for tablet a94d68802b974ed1904134c0955f0c7c (DEFAULT_TABLE table=test-table [id=204146b89d014eaa80358eb04cd8e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:23.639928 2419 tablet_service.cc:1467] Processing CreateTablet for tablet a94d68802b974ed1904134c0955f0c7c (DEFAULT_TABLE table=test-table [id=204146b89d014eaa80358eb04cd8e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:23.640021 2558 tablet_service.cc:1467] Processing CreateTablet for tablet a94d68802b974ed1904134c0955f0c7c (DEFAULT_TABLE table=test-table [id=204146b89d014eaa80358eb04cd8e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:23.640223 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94d68802b974ed1904134c0955f0c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:23.640344 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94d68802b974ed1904134c0955f0c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:23.640372 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94d68802b974ed1904134c0955f0c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:23.644955 2738 tablet_bootstrap.cc:492] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea: Bootstrap starting.
I20260927 22:51:23.646322 2738 tablet_bootstrap.cc:654] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:23.647189 2740 tablet_bootstrap.cc:492] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c: Bootstrap starting.
I20260927 22:51:23.649211 2740 tablet_bootstrap.cc:654] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:23.649466 2738 tablet_bootstrap.cc:492] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea: No bootstrap required, opened a new log
I20260927 22:51:23.649623 2738 ts_tablet_manager.cc:1397] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 22:51:23.650790 2738 raft_consensus.cc:348] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.651008 2738 raft_consensus.cc:374] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:23.651104 2738 raft_consensus.cc:729] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f76d17dbf9945388d5d9d18f5bf1dea, State: Initialized, Role: FOLLOWER
I20260927 22:51:23.651335 2738 consensus_queue.cc:260] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.651522 2740 tablet_bootstrap.cc:492] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c: No bootstrap required, opened a new log
I20260927 22:51:23.651633 2740 ts_tablet_manager.cc:1397] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 22:51:23.651944 2738 ts_tablet_manager.cc:1428] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 22:51:23.652136 2739 tablet_bootstrap.cc:492] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445: Bootstrap starting.
I20260927 22:51:23.652400 2740 raft_consensus.cc:348] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.652648 2740 raft_consensus.cc:374] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:23.652755 2740 raft_consensus.cc:729] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a239314ff74bcb8875449c1e1d170c, State: Initialized, Role: FOLLOWER
I20260927 22:51:23.653019 2740 consensus_queue.cc:260] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.653873 2739 tablet_bootstrap.cc:654] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:23.653896 2740 ts_tablet_manager.cc:1428] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:51:23.655922 2739 tablet_bootstrap.cc:492] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445: No bootstrap required, opened a new log
I20260927 22:51:23.656056 2739 ts_tablet_manager.cc:1397] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 22:51:23.656821 2739 raft_consensus.cc:348] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.657012 2739 raft_consensus.cc:374] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:23.657104 2739 raft_consensus.cc:729] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd12d0edaba44dfeaa978c3c60c60445, State: Initialized, Role: FOLLOWER
I20260927 22:51:23.657313 2739 consensus_queue.cc:260] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.658089 2739 ts_tablet_manager.cc:1428] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 22:51:23.662583 2745 raft_consensus.cc:482] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:23.662720 2745 raft_consensus.cc:504] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.664362 2745 leader_election.cc:288] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1:38493), a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985)
I20260927 22:51:23.674146 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.665148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45985 (local address 127.0.97.3:60379)
0927 22:51:23.668105 (+ 2957us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:23.668120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:23.668255 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:23.668617 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:23.668621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:23.668641 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:23.668841 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:23.668848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.670128 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.670133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:23.672061 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:23.672072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.673299 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.673306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:23.673494 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.673532 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:51:23.673689 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 22:51:23.673812 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2837,"mutex_wait_us":1,"thread_start_us":2727,"threads_started":1}
I20260927 22:51:23.674173 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.665435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60379 (local address 127.0.97.2:45985)
0927 22:51:23.666011 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:23.666020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:23.668200 (+ 2180us) server_negotiation.cc:407] Connection header received
0927 22:51:23.668389 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:23.668397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:23.668479 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:23.668585 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:23.668993 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.670001 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.672208 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.673082 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.673185 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.673283 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:23.673732 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 22:51:23.673865 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":295,"threads_started":1}
I20260927 22:51:23.674497 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.664629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45045 (local address 127.0.97.1:38493)
0927 22:51:23.668744 (+ 4115us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:23.668750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:23.668769 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:51:23.668829 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:23.668832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:23.668879 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:23.668994 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:23.669499 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.670514 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.672519 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.673394 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.673521 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.673665 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:23.674022 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 22:51:23.674170 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3987,"thread_start_us":3840,"threads_started":1}
I20260927 22:51:23.674552 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.664537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38493 (local address 127.0.97.3:45045)
0927 22:51:23.665113 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:23.665156 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:23.665322 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:23.669001 (+ 3679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:23.669004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:23.669020 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:23.669295 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:23.669303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.670636 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.670644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:23.672345 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:23.672362 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.673628 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.673636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:23.673854 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.673888 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:51:23.673982 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:51:23.674105 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":21,"thread_start_us":243,"threads_started":1}
I20260927 22:51:23.675074 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94d68802b974ed1904134c0955f0c7c" candidate_uuid: "bd12d0edaba44dfeaa978c3c60c60445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a239314ff74bcb8875449c1e1d170c" is_pre_election: true
I20260927 22:51:23.675077 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94d68802b974ed1904134c0955f0c7c" candidate_uuid: "bd12d0edaba44dfeaa978c3c60c60445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" is_pre_election: true
I20260927 22:51:23.675292 2439 raft_consensus.cc:2454] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd12d0edaba44dfeaa978c3c60c60445 in term 0.
I20260927 22:51:23.675717 2305 raft_consensus.cc:2454] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd12d0edaba44dfeaa978c3c60c60445 in term 0.
I20260927 22:51:23.676016 2509 leader_election.cc:302] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a239314ff74bcb8875449c1e1d170c, bd12d0edaba44dfeaa978c3c60c60445; no voters:
I20260927 22:51:23.676415 2745 raft_consensus.cc:2781] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:23.676568 2745 raft_consensus.cc:482] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:23.676685 2745 raft_consensus.cc:3037] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:23.679088 2745 raft_consensus.cc:504] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.679631 2745 leader_election.cc:288] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [CANDIDATE]: Term 1 election: Requested vote from peers 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1:38493), a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985)
I20260927 22:51:23.680218 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94d68802b974ed1904134c0955f0c7c" candidate_uuid: "bd12d0edaba44dfeaa978c3c60c60445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a239314ff74bcb8875449c1e1d170c"
I20260927 22:51:23.680092 2305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94d68802b974ed1904134c0955f0c7c" candidate_uuid: "bd12d0edaba44dfeaa978c3c60c60445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea"
I20260927 22:51:23.680418 2439 raft_consensus.cc:3037] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:23.680433 2305 raft_consensus.cc:3037] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:23.682503 2305 raft_consensus.cc:2454] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd12d0edaba44dfeaa978c3c60c60445 in term 1.
I20260927 22:51:23.682514 2439 raft_consensus.cc:2454] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd12d0edaba44dfeaa978c3c60c60445 in term 1.
I20260927 22:51:23.682843 2509 leader_election.cc:302] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f76d17dbf9945388d5d9d18f5bf1dea, bd12d0edaba44dfeaa978c3c60c60445; no voters:
I20260927 22:51:23.683099 2745 raft_consensus.cc:2781] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:23.683542 2745 raft_consensus.cc:686] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 1 LEADER]: Becoming Leader. State: Replica: bd12d0edaba44dfeaa978c3c60c60445, State: Running, Role: LEADER
I20260927 22:51:23.683897 2745 consensus_queue.cc:237] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.687180 2143 catalog_manager.cc:5495] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bd12d0edaba44dfeaa978c3c60c60445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } health_report { overall_health: HEALTHY } } }
I20260927 22:51:23.713966 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.708518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.0.97.3:39163)
0927 22:51:23.709282 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:23.709287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:23.709304 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:51:23.709375 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:23.709378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:23.709442 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:23.709573 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:23.709962 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.710709 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.711940 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.712316 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.712407 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.712776 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:23.713321 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:23.713426 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:23.713515 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 22:51:23.713572 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":544,"threads_started":1}
I20260927 22:51:23.727411 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.722824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54705 (local address 127.0.97.1:38493)
0927 22:51:23.723018 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:23.723022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:23.723147 (+ 125us) server_negotiation.cc:407] Connection header received
0927 22:51:23.723272 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:23.723276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:23.723341 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:23.723410 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:23.724180 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.724830 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.726105 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.726487 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.726575 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.726911 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:23.726996 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:23.727058 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:23.727172 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 22:51:23.727225 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:23.727519 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.722546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38493 (local address 127.0.97.3:54705)
0927 22:51:23.723063 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:23.723078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:23.723180 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:23.723435 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:23.723437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:23.723446 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:51:23.724064 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:23.724071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.724934 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.724937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:23.725993 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:23.726003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.726620 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.726624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:23.726732 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.726775 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:51:23.727076 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:51:23.727082 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:51:23.727140 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 22:51:23.727223 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
I20260927 22:51:23.734611 2439 raft_consensus.cc:1261] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Refusing update from remote peer bd12d0edaba44dfeaa978c3c60c60445: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:23.734629 2305 raft_consensus.cc:1261] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Refusing update from remote peer bd12d0edaba44dfeaa978c3c60c60445: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:23.735147 2750 consensus_queue.cc:1035] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:23.735492 2745 consensus_queue.cc:1035] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:23.764115 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.758679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52645 (local address 127.0.97.3:39163)
0927 22:51:23.758822 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:23.758827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:23.759333 (+ 506us) server_negotiation.cc:407] Connection header received
0927 22:51:23.759523 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:23.759528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:23.759595 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:23.759687 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:23.760324 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.761073 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.762507 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.762909 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.763047 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.763430 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:23.763519 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:23.763611 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:23.763747 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 22:51:23.763825 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:51:23.764209 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.758667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39163 (local address 127.0.97.1:52645)
0927 22:51:23.759229 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:23.759244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:23.759404 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:23.759835 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:23.759840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:23.759852 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:51:23.760153 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:23.760164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.761200 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.761207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:23.762370 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:23.762382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.763079 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.763085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:23.763227 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.763268 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:51:23.763618 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:51:23.763631 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 22:51:23.763728 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:51:23.763850 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":331,"threads_started":1}
I20260927 22:51:23.790474 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:23.790714 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:23.793682 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f: Sending DeleteTablet for 3 replicas of tablet a94d68802b974ed1904134c0955f0c7c
I20260927 22:51:23.794440 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet a94d68802b974ed1904134c0955f0c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57370
I20260927 22:51:23.794565 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet a94d68802b974ed1904134c0955f0c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51736
I20260927 22:51:23.794699 2727 tablet_replica.cc:330] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea: stopping tablet replica
I20260927 22:51:23.794785 2728 tablet_replica.cc:330] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c: stopping tablet replica
I20260927 22:51:23.794747 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet a94d68802b974ed1904134c0955f0c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33676
I20260927 22:51:23.794893 2727 raft_consensus.cc:2229] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:23.794958 2728 raft_consensus.cc:2229] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:23.794998 2729 tablet_replica.cc:330] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445: stopping tablet replica
I20260927 22:51:23.795056 2727 raft_consensus.cc:2258] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:23.795107 2728 raft_consensus.cc:2258] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:23.795174 2729 raft_consensus.cc:2229] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:23.795622 2729 raft_consensus.cc:2258] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:23.796133 2728 ts_tablet_manager.cc:1905] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:23.796725 2729 ts_tablet_manager.cc:1905] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:23.797015 2727 ts_tablet_manager.cc:1905] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:23.803066 2728 ts_tablet_manager.cc:1918] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:51:23.803210 2728 log.cc:1196] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/a94d68802b974ed1904134c0955f0c7c
I20260927 22:51:23.803567 2727 ts_tablet_manager.cc:1918] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:51:23.803624 2728 ts_tablet_manager.cc:1939] T a94d68802b974ed1904134c0955f0c7c P a2a239314ff74bcb8875449c1e1d170c: Deleting consensus metadata
I20260927 22:51:23.803754 2727 log.cc:1196] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/a94d68802b974ed1904134c0955f0c7c
I20260927 22:51:23.804256 2727 ts_tablet_manager.cc:1939] T a94d68802b974ed1904134c0955f0c7c P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting consensus metadata
I20260927 22:51:23.804392 2729 ts_tablet_manager.cc:1918] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:51:23.804616 2729 log.cc:1196] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/a94d68802b974ed1904134c0955f0c7c
I20260927 22:51:23.805238 2729 ts_tablet_manager.cc:1939] T a94d68802b974ed1904134c0955f0c7c P bd12d0edaba44dfeaa978c3c60c60445: Deleting consensus metadata
I20260927 22:51:23.805680 2128 catalog_manager.cc:4841] TS 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1:38493): tablet a94d68802b974ed1904134c0955f0c7c (table test-table [id=204146b89d014eaa80358eb04cd8e176]) successfully deleted
I20260927 22:51:23.805967 2128 catalog_manager.cc:4841] TS a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985): tablet a94d68802b974ed1904134c0955f0c7c (table test-table [id=204146b89d014eaa80358eb04cd8e176]) successfully deleted
I20260927 22:51:23.805882 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:51:23.806238 2131 catalog_manager.cc:4841] TS bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163): tablet a94d68802b974ed1904134c0955f0c7c (table test-table [id=204146b89d014eaa80358eb04cd8e176]) successfully deleted
W20260927 22:51:23.806759 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:23.813032 2280 tablet_service.cc:1467] Processing CreateTablet for tablet a667e6a801db4ec8a3744422b9f30c03 (DEFAULT_TABLE table=test-table [id=8b7eb80a9b734bbdb280cec74065acfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:23.813427 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a667e6a801db4ec8a3744422b9f30c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:23.813496 2419 tablet_service.cc:1467] Processing CreateTablet for tablet a667e6a801db4ec8a3744422b9f30c03 (DEFAULT_TABLE table=test-table [id=8b7eb80a9b734bbdb280cec74065acfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:23.813952 2558 tablet_service.cc:1467] Processing CreateTablet for tablet a667e6a801db4ec8a3744422b9f30c03 (DEFAULT_TABLE table=test-table [id=8b7eb80a9b734bbdb280cec74065acfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:23.814322 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a667e6a801db4ec8a3744422b9f30c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:23.814352 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a667e6a801db4ec8a3744422b9f30c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:23.819466 2738 tablet_bootstrap.cc:492] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea: Bootstrap starting.
I20260927 22:51:23.820806 2740 tablet_bootstrap.cc:492] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c: Bootstrap starting.
I20260927 22:51:23.821391 2738 tablet_bootstrap.cc:654] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:23.822415 2740 tablet_bootstrap.cc:654] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:23.823014 2739 tablet_bootstrap.cc:492] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445: Bootstrap starting.
I20260927 22:51:23.825040 2739 tablet_bootstrap.cc:654] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:23.830735 2740 tablet_bootstrap.cc:492] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c: No bootstrap required, opened a new log
I20260927 22:51:23.830916 2740 ts_tablet_manager.cc:1397] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260927 22:51:23.831564 2738 tablet_bootstrap.cc:492] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea: No bootstrap required, opened a new log
I20260927 22:51:23.831779 2738 ts_tablet_manager.cc:1397] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20260927 22:51:23.831625 2740 raft_consensus.cc:348] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.831873 2740 raft_consensus.cc:374] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:23.831979 2740 raft_consensus.cc:729] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a239314ff74bcb8875449c1e1d170c, State: Initialized, Role: FOLLOWER
I20260927 22:51:23.832190 2739 tablet_bootstrap.cc:492] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445: No bootstrap required, opened a new log
I20260927 22:51:23.832386 2739 ts_tablet_manager.cc:1397] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260927 22:51:23.832262 2740 consensus_queue.cc:260] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.833140 2739 raft_consensus.cc:348] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.833423 2739 raft_consensus.cc:374] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:23.833388 2738 raft_consensus.cc:348] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.833544 2739 raft_consensus.cc:729] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd12d0edaba44dfeaa978c3c60c60445, State: Initialized, Role: FOLLOWER
I20260927 22:51:23.833730 2738 raft_consensus.cc:374] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:23.833863 2738 raft_consensus.cc:729] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f76d17dbf9945388d5d9d18f5bf1dea, State: Initialized, Role: FOLLOWER
I20260927 22:51:23.833832 2739 consensus_queue.cc:260] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.834190 2738 consensus_queue.cc:260] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.834416 2739 ts_tablet_manager.cc:1428] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:51:23.835160 2740 ts_tablet_manager.cc:1428] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:51:23.835634 2738 ts_tablet_manager.cc:1428] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260927 22:51:23.857468 2714 raft_consensus.cc:482] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:23.857687 2714 raft_consensus.cc:504] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.858489 2714 leader_election.cc:288] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985), bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163)
I20260927 22:51:23.858922 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a667e6a801db4ec8a3744422b9f30c03" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd12d0edaba44dfeaa978c3c60c60445" is_pre_election: true
I20260927 22:51:23.858953 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a667e6a801db4ec8a3744422b9f30c03" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a239314ff74bcb8875449c1e1d170c" is_pre_election: true
I20260927 22:51:23.859172 2439 raft_consensus.cc:2454] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 0.
I20260927 22:51:23.859181 2577 raft_consensus.cc:2454] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 0.
I20260927 22:51:23.859611 2231 leader_election.cc:302] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f76d17dbf9945388d5d9d18f5bf1dea, a2a239314ff74bcb8875449c1e1d170c; no voters:
I20260927 22:51:23.859889 2714 raft_consensus.cc:2781] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:23.860406 2714 raft_consensus.cc:482] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:23.860513 2714 raft_consensus.cc:3037] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:23.862735 2714 raft_consensus.cc:504] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.863368 2714 leader_election.cc:288] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 election: Requested vote from peers a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985), bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163)
I20260927 22:51:23.863754 2439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a667e6a801db4ec8a3744422b9f30c03" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a239314ff74bcb8875449c1e1d170c"
I20260927 22:51:23.863863 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a667e6a801db4ec8a3744422b9f30c03" candidate_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd12d0edaba44dfeaa978c3c60c60445"
I20260927 22:51:23.863953 2439 raft_consensus.cc:3037] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:23.864061 2577 raft_consensus.cc:3037] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:23.866871 2439 raft_consensus.cc:2454] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 1.
I20260927 22:51:23.867101 2577 raft_consensus.cc:2454] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f76d17dbf9945388d5d9d18f5bf1dea in term 1.
I20260927 22:51:23.867267 2231 leader_election.cc:302] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f76d17dbf9945388d5d9d18f5bf1dea, a2a239314ff74bcb8875449c1e1d170c; no voters:
I20260927 22:51:23.867548 2714 raft_consensus.cc:2781] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:23.867710 2714 raft_consensus.cc:686] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 LEADER]: Becoming Leader. State: Replica: 4f76d17dbf9945388d5d9d18f5bf1dea, State: Running, Role: LEADER
I20260927 22:51:23.867972 2714 consensus_queue.cc:237] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } }
I20260927 22:51:23.870687 2143 catalog_manager.cc:5495] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f76d17dbf9945388d5d9d18f5bf1dea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 } health_report { overall_health: UNKNOWN } } }
I20260927 22:51:23.891283 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:23.886174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.0.97.1:38493)
0927 22:51:23.886332 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:23.886338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:23.886421 (+ 83us) server_negotiation.cc:407] Connection header received
0927 22:51:23.886577 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:23.886582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:23.886658 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:23.886756 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:23.887047 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.888180 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.889447 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:23.890111 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:23.890235 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:23.890570 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:23.890697 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:23.890784 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:23.890912 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 22:51:23.890993 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:51:23.900599 2578 raft_consensus.cc:1261] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Refusing update from remote peer 4f76d17dbf9945388d5d9d18f5bf1dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:23.900624 2439 raft_consensus.cc:1261] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Refusing update from remote peer 4f76d17dbf9945388d5d9d18f5bf1dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:23.901134 2654 consensus_queue.cc:1035] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a239314ff74bcb8875449c1e1d170c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:23.901466 2714 consensus_queue.cc:1035] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd12d0edaba44dfeaa978c3c60c60445" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:51:23.923378 2707 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:51:25.497747 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:25.497956 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:25.502045 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6af96a0326194af48ad02190f652a78f: Sending DeleteTablet for 3 replicas of tablet a667e6a801db4ec8a3744422b9f30c03
I20260927 22:51:25.502763 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet a667e6a801db4ec8a3744422b9f30c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57370
I20260927 22:51:25.502955 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet a667e6a801db4ec8a3744422b9f30c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51736
I20260927 22:51:25.503007 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet a667e6a801db4ec8a3744422b9f30c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33676
I20260927 22:51:25.503748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 2215
I20260927 22:51:25.512137 2789 tablet_replica.cc:330] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea: stopping tablet replica
I20260927 22:51:25.512660 2791 tablet_replica.cc:330] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445: stopping tablet replica
I20260927 22:51:25.512804 2790 tablet_replica.cc:330] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c: stopping tablet replica
I20260927 22:51:25.512883 2791 raft_consensus.cc:2229] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:25.512998 2789 raft_consensus.cc:2229] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:25.513127 2791 raft_consensus.cc:2258] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:25.513141 2790 raft_consensus.cc:2229] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:25.513405 2789 raft_consensus.cc:2258] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:25.513460 2790 raft_consensus.cc:2258] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:25.514199 2791 ts_tablet_manager.cc:1905] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:25.514703 2790 ts_tablet_manager.cc:1905] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:25.515095 2789 ts_tablet_manager.cc:1905] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:25.520656 2791 ts_tablet_manager.cc:1918] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:51:25.520855 2791 log.cc:1196] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/a667e6a801db4ec8a3744422b9f30c03
I20260927 22:51:25.521307 2791 ts_tablet_manager.cc:1939] T a667e6a801db4ec8a3744422b9f30c03 P bd12d0edaba44dfeaa978c3c60c60445: Deleting consensus metadata
I20260927 22:51:25.521844 2790 ts_tablet_manager.cc:1918] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:51:25.522020 2790 log.cc:1196] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/a667e6a801db4ec8a3744422b9f30c03
I20260927 22:51:25.522265 2789 ts_tablet_manager.cc:1918] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:51:25.522439 2789 log.cc:1196] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/a667e6a801db4ec8a3744422b9f30c03
I20260927 22:51:25.522433 2131 catalog_manager.cc:4841] TS bd12d0edaba44dfeaa978c3c60c60445 (127.0.97.3:39163): tablet a667e6a801db4ec8a3744422b9f30c03 (table test-table [id=8b7eb80a9b734bbdb280cec74065acfe]) successfully deleted
Sep 27 22:51:25 dist-test-slave-df53 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549483, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:25.522506 2790 ts_tablet_manager.cc:1939] T a667e6a801db4ec8a3744422b9f30c03 P a2a239314ff74bcb8875449c1e1d170c: Deleting consensus metadata
I20260927 22:51:25.522966 2789 ts_tablet_manager.cc:1939] T a667e6a801db4ec8a3744422b9f30c03 P 4f76d17dbf9945388d5d9d18f5bf1dea: Deleting consensus metadata
I20260927 22:51:25.523593 2128 catalog_manager.cc:4841] TS a2a239314ff74bcb8875449c1e1d170c (127.0.97.2:45985): tablet a667e6a801db4ec8a3744422b9f30c03 (table test-table [id=8b7eb80a9b734bbdb280cec74065acfe]) successfully deleted
I20260927 22:51:25.523932 2128 catalog_manager.cc:4841] TS 4f76d17dbf9945388d5d9d18f5bf1dea (127.0.97.1:38493): tablet a667e6a801db4ec8a3744422b9f30c03 (table test-table [id=8b7eb80a9b734bbdb280cec74065acfe]) successfully deleted
I20260927 22:51:25.529047 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.505977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.0.97.1:38493)
0927 22:51:25.512474 (+ 6497us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:25.512481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:25.512507 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:51:25.512575 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:25.512582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:25.512668 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:25.513749 (+ 1081us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:25.514412 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:25.515469 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:25.517240 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:25.517909 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:25.518041 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:25.523490 (+ 5449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:25.523507 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:25.523535 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:25.523568 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:25.526484 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:25.527123 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:25.527131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:25.527133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:25.527241 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:25.527614 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:25.527623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:25.527625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:25.527937 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:25.528243 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:25.528460 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 22:51:25.528658 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6373,"thread_start_us":6213,"threads_started":1}
W20260927 22:51:25.529683 2345 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44306
W20260927 22:51:25.530352 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 2215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:51:25.530565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 2215
I20260927 22:51:25.564188 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 2354
Sep 27 22:51:25 dist-test-slave-df53 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549483, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:25.581970 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.564982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.0.97.2:45985)
0927 22:51:25.565801 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:25.565806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:25.565820 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:51:25.565869 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:25.565872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:25.565958 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:25.566096 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:25.567419 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:25.568258 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:25.569505 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:25.570045 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:25.570193 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:25.576093 (+ 5900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:25.576112 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:25.576136 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:25.576177 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:25.579246 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:25.579924 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:25.579936 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:25.579948 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:25.580162 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:25.580507 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:25.580511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:25.580513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:25.580977 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:25.581213 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:25.581501 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 22:51:25.581731 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":580,"threads_started":1}
W20260927 22:51:25.582628 2484 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56742
W20260927 22:51:25.583227 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 2354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:51:25.583446 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 2354
I20260927 22:51:25.605662 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 2493
Sep 27 22:51:25 dist-test-slave-df53 krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549483, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:51:25.619802 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:25.606348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.0.97.3:39163)
0927 22:51:25.606905 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:25.606909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:25.606923 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:51:25.606973 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:25.606976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:25.607029 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:25.607163 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:25.608412 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:25.609133 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:25.610482 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:25.610878 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:25.611006 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:25.615061 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:25.615074 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:25.615089 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:25.615120 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:25.617576 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:25.618204 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:25.618211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:25.618213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:25.618329 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:25.618625 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:25.618628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:25.618630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:25.618935 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:25.619160 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:25.619382 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 22:51:25.619529 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":266,"threads_started":1}
W20260927 22:51:25.620500 2623 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55226
W20260927 22:51:25.621114 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 2493: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:51:25.621250 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 2493
I20260927 22:51:25.642148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 2112
I20260927 22:51:25.643174 2184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:25.757287 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 2112
2026-09-27T22:51:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7878 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:25 dist-test-slave-df53 krb5kdc[2799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:25 dist-test-slave-df53 krb5kdc[2799](info): set up 2 sockets
Sep 27 22:51:25 dist-test-slave-df53 krb5kdc[2799](info): commencing operation
krb5kdc: starting...
W20260927 22:51:26.841890 388 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:26 dist-test-slave-df53 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:27.012945 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41253
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:27.310958 2812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:27.311225 2812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:27.311311 2812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:27.318161 2812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:27.318254 2812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:27.318301 2812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:27.318358 2812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:27.318400 2812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:27.327587 2812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:27.329116 2812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:27.330762 2812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:27.341141 2817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:27.341966 2818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:27.343307 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:27.343545 2812 server_base.cc:1034] running on GCE node
I20260927 22:51:27.344259 2812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:27.345687 2812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:27.346912 2812 hybrid_clock.cc:648] HybridClock initialized: now 1790549487346854 us; error 47 us; skew 500 ppm
I20260927 22:51:27.347235 2812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:27 dist-test-slave-df53 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:27.350198 2812 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:27.351550 2812 webserver.cc:460] Webserver started at http://127.0.97.62:34487/ using document root <none> and password file <none>
I20260927 22:51:27.352205 2812 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:27.352347 2812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:27.352649 2812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:27.355477 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "93f77e7927ff49a3a5db170005f2336b"
format_stamp: "Formatted at 2026-09-27 22:51:27 on dist-test-slave-df53"
server_key: "05c62955eeeba5a57e51fe127957d86a"
server_key_iv: "3fd22bfee7004b7d2c1d6f8c31f9bd73"
server_key_version: "encryptionkey@0"
I20260927 22:51:27.356210 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "93f77e7927ff49a3a5db170005f2336b"
format_stamp: "Formatted at 2026-09-27 22:51:27 on dist-test-slave-df53"
server_key: "05c62955eeeba5a57e51fe127957d86a"
server_key_iv: "3fd22bfee7004b7d2c1d6f8c31f9bd73"
server_key_version: "encryptionkey@0"
I20260927 22:51:27.360952 2812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:51:27.364195 2827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:27.365409 2812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 22:51:27.365664 2812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "93f77e7927ff49a3a5db170005f2336b"
format_stamp: "Formatted at 2026-09-27 22:51:27 on dist-test-slave-df53"
server_key: "05c62955eeeba5a57e51fe127957d86a"
server_key_iv: "3fd22bfee7004b7d2c1d6f8c31f9bd73"
server_key_version: "encryptionkey@0"
I20260927 22:51:27.365861 2812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:27.385499 2812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:27.389371 2812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:27.389701 2812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:27.407658 2812 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41253
I20260927 22:51:27.407691 2878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41253 every 8 connection(s)
I20260927 22:51:27.409004 2812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:27.412685 2879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:27.417040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 2812
I20260927 22:51:27.417266 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:27.417977 388 external_mini_cluster.cc:1428] Setting key 2fec037fc4c18f8f547bd438537df240
I20260927 22:51:27.422878 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b: Bootstrap starting.
I20260927 22:51:27.426460 2879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:27.427668 2879 log.cc:824] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:27.430740 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b: No bootstrap required, opened a new log
Sep 27 22:51:27 dist-test-slave-df53 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:27.438385 2879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f77e7927ff49a3a5db170005f2336b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41253 } }
I20260927 22:51:27.438710 2879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:27.438813 2879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f77e7927ff49a3a5db170005f2336b, State: Initialized, Role: FOLLOWER
I20260927 22:51:27.439483 2879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f77e7927ff49a3a5db170005f2336b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41253 } }
I20260927 22:51:27.439716 2879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:27.439862 2879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:27.440091 2879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:27.440922 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:27.419948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.0.97.62:41253)
0927 22:51:27.420835 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:27.420848 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:27.420890 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:51:27.422605 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:27.422627 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:27.423015 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:27.423341 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:27.425173 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:27.426521 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:27.428341 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:27.429271 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:27.429405 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:27.433909 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:27.433917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:27.433933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:27.433972 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:27.436992 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:27.437692 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:27.437706 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:27.437712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:27.437865 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:27.438223 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:27.438229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:27.438233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:27.438755 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:27.439068 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:27.439493 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 22:51:27.439904 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":437,"threads_started":1}
I20260927 22:51:27.442566 2879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f77e7927ff49a3a5db170005f2336b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41253 } }
I20260927 22:51:27.443104 2879 leader_election.cc:302] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93f77e7927ff49a3a5db170005f2336b; no voters:
I20260927 22:51:27.443763 2879 leader_election.cc:288] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:27.443949 2884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:27.444587 2884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [term 1 LEADER]: Becoming Leader. State: Replica: 93f77e7927ff49a3a5db170005f2336b, State: Running, Role: LEADER
I20260927 22:51:27.445130 2884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f77e7927ff49a3a5db170005f2336b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41253 } }
I20260927 22:51:27.445783 2879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:27.448406 2886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93f77e7927ff49a3a5db170005f2336b. Latest consensus state: current_term: 1 leader_uuid: "93f77e7927ff49a3a5db170005f2336b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f77e7927ff49a3a5db170005f2336b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41253 } } }
I20260927 22:51:27.448738 2886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:27.450296 2895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:27.450186 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93f77e7927ff49a3a5db170005f2336b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f77e7927ff49a3a5db170005f2336b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41253 } } }
I20260927 22:51:27.451391 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:27.454658 2895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:27.462407 2895 catalog_manager.cc:1347] Generated new cluster ID: 72f77ae4ea8b46ba8cd45a27fc939961
I20260927 22:51:27.462519 2895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:27.473426 2895 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:27.474704 2895 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:27.488379 2895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93f77e7927ff49a3a5db170005f2336b: Generated new TSK 0
I20260927 22:51:27.489164 2895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:27.534682 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41253
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:27.845243 2905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:27.845491 2905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:27.845705 2905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:27.852974 2905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:27.853063 2905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:27.853266 2905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:27.862535 2905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:27.863724 2905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:27.864989 2905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:27.877784 2913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:27.877840 2910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:27.877862 2911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:27.878486 2905 server_base.cc:1034] running on GCE node
I20260927 22:51:27.879037 2905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:27.879913 2905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:27.881145 2905 hybrid_clock.cc:648] HybridClock initialized: now 1790549487881094 us; error 44 us; skew 500 ppm
I20260927 22:51:27.881453 2905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:27 dist-test-slave-df53 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:27.884459 2905 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:27.885919 2905 webserver.cc:460] Webserver started at http://127.0.97.1:35721/ using document root <none> and password file <none>
I20260927 22:51:27.886574 2905 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:27.886715 2905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:27.887020 2905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:27.889889 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "f6535349ba074833aa5278e624a5ecb2"
format_stamp: "Formatted at 2026-09-27 22:51:27 on dist-test-slave-df53"
server_key: "e3ed79ee62a207df803d137314a11342"
server_key_iv: "a681fb2dfae95e770640923ca9daacb8"
server_key_version: "encryptionkey@0"
I20260927 22:51:27.890668 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f6535349ba074833aa5278e624a5ecb2"
format_stamp: "Formatted at 2026-09-27 22:51:27 on dist-test-slave-df53"
server_key: "e3ed79ee62a207df803d137314a11342"
server_key_iv: "a681fb2dfae95e770640923ca9daacb8"
server_key_version: "encryptionkey@0"
I20260927 22:51:27.895646 2905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:51:27.899030 2920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:27.900349 2905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:27.900525 2905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "f6535349ba074833aa5278e624a5ecb2"
format_stamp: "Formatted at 2026-09-27 22:51:27 on dist-test-slave-df53"
server_key: "e3ed79ee62a207df803d137314a11342"
server_key_iv: "a681fb2dfae95e770640923ca9daacb8"
server_key_version: "encryptionkey@0"
I20260927 22:51:27.900727 2905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:27.921367 2905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:27.925153 2905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:27.925458 2905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:27.926385 2905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:27.927644 2905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:27.927752 2905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:27.927898 2905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:27.927970 2905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:27.959775 2905 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40307
I20260927 22:51:27.959856 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40307 every 8 connection(s)
I20260927 22:51:27.960987 2905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:27.970660 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 2905
I20260927 22:51:27.970880 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:27.971513 388 external_mini_cluster.cc:1428] Setting key c9c753c448882df5aa1739593e8b3968
Sep 27 22:51:27 dist-test-slave-df53 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:27.980551 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:27.963437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47039 (local address 127.0.97.62:41253)
0927 22:51:27.963834 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:27.963839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:27.964808 (+ 969us) server_negotiation.cc:407] Connection header received
0927 22:51:27.966683 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:27.966688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:27.966750 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:27.966848 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:27.968284 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:27.969092 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:27.971107 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:27.971497 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:27.971638 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:27.975217 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:27.975222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:27.975225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:27.975254 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:27.977687 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:27.978385 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:27.978390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:27.978394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:27.978514 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:27.978923 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:27.978928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:27.978932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:27.979186 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:27.979345 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:27.979988 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 22:51:27.980203 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":205,"threads_started":1}
I20260927 22:51:27.981420 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:27.963990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41253 (local address 127.0.97.1:47039)
0927 22:51:27.964658 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:27.964694 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:27.966442 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:27.967085 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:27.967094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:27.967568 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:27.968105 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:27.968126 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:27.969256 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:27.969261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:27.970969 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:27.970983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:27.971723 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:27.971729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:27.972012 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:27.972724 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:27.972747 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:27.974991 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:27.977820 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:27.977828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:27.977843 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:27.978259 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:27.978622 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:27.978628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:27.978631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:27.978806 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:27.979321 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:27.979328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:27.979657 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 22:51:27.979903 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 22:51:27.980373 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":266,"threads_started":1}
I20260927 22:51:27.982920 3033 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41253
I20260927 22:51:27.983224 3033 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:27.984030 3033 heartbeater.cc:511] Master 127.0.97.62:41253 requested a full tablet report, sending...
I20260927 22:51:27.986470 2844 ts_manager.cc:194] Registered new tserver with Master: f6535349ba074833aa5278e624a5ecb2 (127.0.97.1:40307)
I20260927 22:51:27.988682 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47039
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:28.010874 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41253
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:28.317329 3039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:28.317627 3039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:28.317742 3039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:28.324695 3039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:28.324785 3039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:28.324904 3039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:28.335516 3039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:28.336632 3039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:28.337901 3039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:28.350561 3044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:28.350653 3047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:28.350602 3045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:28.351642 3039 server_base.cc:1034] running on GCE node
I20260927 22:51:28.352207 3039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:28.353053 3039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:28.354238 3039 hybrid_clock.cc:648] HybridClock initialized: now 1790549488354201 us; error 32 us; skew 500 ppm
I20260927 22:51:28.354533 3039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:28 dist-test-slave-df53 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:28.357465 3039 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:28.358850 3039 webserver.cc:460] Webserver started at http://127.0.97.2:38921/ using document root <none> and password file <none>
I20260927 22:51:28.359441 3039 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:28.359546 3039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:28.359778 3039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:28.362504 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "b2d03aa8069f4dfc96b972d1eefab879"
format_stamp: "Formatted at 2026-09-27 22:51:28 on dist-test-slave-df53"
server_key: "a83050edb16aec3f8a206ae789f45bfa"
server_key_iv: "70b3d94246d3c69d81cb8e179bc7a8e6"
server_key_version: "encryptionkey@0"
I20260927 22:51:28.363205 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b2d03aa8069f4dfc96b972d1eefab879"
format_stamp: "Formatted at 2026-09-27 22:51:28 on dist-test-slave-df53"
server_key: "a83050edb16aec3f8a206ae789f45bfa"
server_key_iv: "70b3d94246d3c69d81cb8e179bc7a8e6"
server_key_version: "encryptionkey@0"
I20260927 22:51:28.368032 3039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:51:28.371402 3054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:28.372665 3039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:28.372828 3039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "b2d03aa8069f4dfc96b972d1eefab879"
format_stamp: "Formatted at 2026-09-27 22:51:28 on dist-test-slave-df53"
server_key: "a83050edb16aec3f8a206ae789f45bfa"
server_key_iv: "70b3d94246d3c69d81cb8e179bc7a8e6"
server_key_version: "encryptionkey@0"
I20260927 22:51:28.373028 3039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:28.387310 3039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:28.391284 3039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:28.391571 3039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:28.392462 3039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:28.393934 3039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:28.394033 3039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:28.394196 3039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:28.394281 3039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:28.427508 3039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34307
I20260927 22:51:28.427561 3166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34307 every 8 connection(s)
I20260927 22:51:28.428699 3039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:28.435885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 3039
I20260927 22:51:28.436087 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:28.436502 388 external_mini_cluster.cc:1428] Setting key 821a7ac79b40c615a00a40cda3de71d0
Sep 27 22:51:28 dist-test-slave-df53 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:28.449733 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:28.431235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46067 (local address 127.0.97.62:41253)
0927 22:51:28.431404 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:28.431408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:28.432557 (+ 1149us) server_negotiation.cc:407] Connection header received
0927 22:51:28.434459 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:28.434465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:28.434554 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:28.434661 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:28.436017 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:28.436817 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:28.438630 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:28.439046 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:28.439175 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:28.442955 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:28.442963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:28.442969 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:28.443010 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:28.446713 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:28.447490 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:28.447498 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:28.447505 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:28.447649 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:28.448068 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:28.448072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:28.448074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:28.448352 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:28.448516 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:28.449162 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 22:51:28.449380 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:51:28.450773 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:28.431716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41253 (local address 127.0.97.2:46067)
0927 22:51:28.432403 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:28.432449 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:28.434209 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:28.434892 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:28.434900 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:28.435351 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:28.435861 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:28.435877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:28.436995 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:28.437002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:28.438496 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:28.438507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:28.439252 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:28.439258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:28.439477 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:28.440111 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:28.440132 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:28.442719 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:28.446859 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:28.446866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:28.446877 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:28.447375 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:28.447751 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:28.447754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:28.447756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:28.447915 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:28.448491 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:28.448499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:28.448804 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 22:51:28.449059 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 22:51:28.449562 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":282,"threads_started":1}
I20260927 22:51:28.452145 3167 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41253
I20260927 22:51:28.452464 3167 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:28.453303 3167 heartbeater.cc:511] Master 127.0.97.62:41253 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:51:28.454793 2844 ts_manager.cc:194] Registered new tserver with Master: b2d03aa8069f4dfc96b972d1eefab879 (127.0.97.2:34307)
I20260927 22:51:28.455559 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:28.477432 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41253
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:28.793375 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:28.793654 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:28.793807 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:28.800715 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:28.800814 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:28.801029 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:28.810395 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:28.811579 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:28.812786 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:28.825119 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:28.825217 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:28.825567 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:28.826071 3172 server_base.cc:1034] running on GCE node
I20260927 22:51:28.826557 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:28.827354 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:28.828552 3172 hybrid_clock.cc:648] HybridClock initialized: now 1790549488828525 us; error 32 us; skew 500 ppm
I20260927 22:51:28.828796 3172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:28 dist-test-slave-df53 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:28.835086 3172 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:28.836380 3172 webserver.cc:460] Webserver started at http://127.0.97.3:33213/ using document root <none> and password file <none>
I20260927 22:51:28.837014 3172 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:28.837137 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:28.837440 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:28.840179 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "6346a046b4e947108737f04490cef617"
format_stamp: "Formatted at 2026-09-27 22:51:28 on dist-test-slave-df53"
server_key: "f7d67f5ac9233a9d5c3fa604ca90ec42"
server_key_iv: "40b747e1f24932351c87ca40afcdc751"
server_key_version: "encryptionkey@0"
I20260927 22:51:28.840919 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6346a046b4e947108737f04490cef617"
format_stamp: "Formatted at 2026-09-27 22:51:28 on dist-test-slave-df53"
server_key: "f7d67f5ac9233a9d5c3fa604ca90ec42"
server_key_iv: "40b747e1f24932351c87ca40afcdc751"
server_key_version: "encryptionkey@0"
I20260927 22:51:28.845716 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.007s
I20260927 22:51:28.849118 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:28.850342 3172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:51:28.850494 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "6346a046b4e947108737f04490cef617"
format_stamp: "Formatted at 2026-09-27 22:51:28 on dist-test-slave-df53"
server_key: "f7d67f5ac9233a9d5c3fa604ca90ec42"
server_key_iv: "40b747e1f24932351c87ca40afcdc751"
server_key_version: "encryptionkey@0"
I20260927 22:51:28.850675 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:28.870539 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:28.874172 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:28.874455 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:28.875337 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:28.876616 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:28.876684 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:28.876809 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:28.876880 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:28.908500 3172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40357
I20260927 22:51:28.908582 3299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40357 every 8 connection(s)
I20260927 22:51:28.909816 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:28.912868 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 3172
I20260927 22:51:28.913066 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:28.913622 388 external_mini_cluster.cc:1428] Setting key ddfc5570e30910b776158c2ee0bac668
Sep 27 22:51:28 dist-test-slave-df53 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:28.928695 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:28.912194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50299 (local address 127.0.97.62:41253)
0927 22:51:28.912359 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:28.912363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:28.913722 (+ 1359us) server_negotiation.cc:407] Connection header received
0927 22:51:28.915999 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:28.916002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:28.916061 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:28.916137 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:28.917685 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:28.918379 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:28.919819 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:28.920194 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:28.920281 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:28.923441 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:28.923446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:28.923449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:28.923477 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:28.925719 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:28.926312 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:28.926315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:28.926317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:28.926419 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:28.926800 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:28.926804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:28.926806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:28.927154 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:28.927287 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:28.928151 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 22:51:28.928346 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:51:28.929507 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:28.912698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41253 (local address 127.0.97.3:50299)
0927 22:51:28.913524 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:28.913609 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:28.915716 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:28.916403 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:28.916412 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:28.916990 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:28.917504 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:28.917519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:28.918502 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:28.918505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:28.919694 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:28.919705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:28.920359 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:28.920364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:28.920539 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:28.921190 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:28.921215 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:28.923262 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:28.925828 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:28.925834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:28.925844 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:28.926212 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:28.926504 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:28.926507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:28.926509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:28.926663 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:28.927303 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:28.927310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:28.927720 (+ 410us) client_negotiation.cc:769] Sending connection context
0927 22:51:28.927997 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 22:51:28.928432 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":347,"threads_started":1}
I20260927 22:51:28.930831 3300 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41253
I20260927 22:51:28.931118 3300 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:28.931749 3300 heartbeater.cc:511] Master 127.0.97.62:41253 requested a full tablet report, sending...
I20260927 22:51:28.933097 2844 ts_manager.cc:194] Registered new tserver with Master: 6346a046b4e947108737f04490cef617 (127.0.97.3:40357)
I20260927 22:51:28.933943 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50299
I20260927 22:51:28.940665 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:51:28.968921 3035 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:51:28.966630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.0.97.62:41253)
0927 22:51:28.966863 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:28.966871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:28.967219 (+ 348us) server_negotiation.cc:407] Connection header received
0927 22:51:28.967421 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:28.967426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:28.967490 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:28.967594 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:28.968748 (+ 1154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37806: BlockingRecv error: recv got EOF from 127.0.0.1:37806 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:51:28.972203 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 2905
W20260927 22:51:28.974810 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 2905: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:40307: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:51:28.974972 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 2905
I20260927 22:51:28.990249 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3039
W20260927 22:51:28.992573 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3039: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:34307: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:51:28.992730 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 3039
I20260927 22:51:29.007296 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3172
W20260927 22:51:29.009984 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3172: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:40357: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:51:29.010102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 3172
I20260927 22:51:29.024605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 2812
I20260927 22:51:29.025758 2874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:29.124840 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 2812
2026-09-27T22:51:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3363 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:29 dist-test-slave-df53 krb5kdc[3317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:29 dist-test-slave-df53 krb5kdc[3317](info): set up 2 sockets
Sep 27 22:51:29 dist-test-slave-df53 krb5kdc[3317](info): commencing operation
krb5kdc: starting...
W20260927 22:51:30.198304 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:30 dist-test-slave-df53 krb5kdc[3317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:30.375113 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35755
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:30.683501 3330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:30.683816 3330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:30.683980 3330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:30.691046 3330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:30.691182 3330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:30.691282 3330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:30.691380 3330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:30.691462 3330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:30.700886 3330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:30.702133 3330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:30.703369 3330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:30.714438 3335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:30.714457 3336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:30.714453 3338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:30.715773 3330 server_base.cc:1034] running on GCE node
I20260927 22:51:30.716485 3330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:30.717765 3330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:30.718982 3330 hybrid_clock.cc:648] HybridClock initialized: now 1790549490718926 us; error 45 us; skew 500 ppm
I20260927 22:51:30.719338 3330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:30 dist-test-slave-df53 krb5kdc[3317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:30.722404 3330 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:30.723688 3330 webserver.cc:460] Webserver started at http://127.0.97.62:43327/ using document root <none> and password file <none>
I20260927 22:51:30.724325 3330 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:30.724478 3330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:30.724785 3330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:30.727566 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "9107fad6107e4c2b9eb65bdde2941fd5"
format_stamp: "Formatted at 2026-09-27 22:51:30 on dist-test-slave-df53"
server_key: "fb8598bcf0d681c0fc83f04d09d10571"
server_key_iv: "80409b5de2f6f25cd11de2c944354deb"
server_key_version: "encryptionkey@0"
I20260927 22:51:30.728322 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "9107fad6107e4c2b9eb65bdde2941fd5"
format_stamp: "Formatted at 2026-09-27 22:51:30 on dist-test-slave-df53"
server_key: "fb8598bcf0d681c0fc83f04d09d10571"
server_key_iv: "80409b5de2f6f25cd11de2c944354deb"
server_key_version: "encryptionkey@0"
I20260927 22:51:30.733019 3330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 22:51:30.736127 3345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:30.737303 3330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 22:51:30.737469 3330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "9107fad6107e4c2b9eb65bdde2941fd5"
format_stamp: "Formatted at 2026-09-27 22:51:30 on dist-test-slave-df53"
server_key: "fb8598bcf0d681c0fc83f04d09d10571"
server_key_iv: "80409b5de2f6f25cd11de2c944354deb"
server_key_version: "encryptionkey@0"
I20260927 22:51:30.737720 3330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:30.755344 3330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:30.759426 3330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:30.759678 3330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:30.777424 3330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35755
I20260927 22:51:30.777456 3396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35755 every 8 connection(s)
I20260927 22:51:30.778719 3330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:30.779771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 3330
I20260927 22:51:30.779985 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:30.780769 388 external_mini_cluster.cc:1428] Setting key d1afb296dafcabead6a9da6723fb2f5b
I20260927 22:51:30.783218 3397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:51:30 dist-test-slave-df53 krb5kdc[3317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:30.795774 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5: Bootstrap starting.
I20260927 22:51:30.799038 3397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:30.800019 3397 log.cc:824] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:30.802635 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5: No bootstrap required, opened a new log
I20260927 22:51:30.802654 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:30.782577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.0.97.62:35755)
0927 22:51:30.783348 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:30.783363 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:30.783410 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:51:30.784967 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:30.784987 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:30.785335 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:30.785759 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:30.787667 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:30.788976 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:30.790709 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:30.791372 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:30.791508 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:30.795843 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:30.795851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:30.795863 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:30.795924 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:30.798848 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:30.799565 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:30.799571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:30.799576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:30.799719 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:30.800084 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:30.800095 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:30.800098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:30.800634 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:30.800933 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:30.801374 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 22:51:30.801756 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":351,"threads_started":1}
I20260927 22:51:30.806308 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:51:30.810210 3397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9107fad6107e4c2b9eb65bdde2941fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260927 22:51:30.810556 3397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:30.810660 3397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9107fad6107e4c2b9eb65bdde2941fd5, State: Initialized, Role: FOLLOWER
I20260927 22:51:30.811228 3397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9107fad6107e4c2b9eb65bdde2941fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260927 22:51:30.811462 3397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:30.811599 3397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:30.811762 3397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:30.813772 3397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9107fad6107e4c2b9eb65bdde2941fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260927 22:51:30.814209 3397 leader_election.cc:302] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9107fad6107e4c2b9eb65bdde2941fd5; no voters:
I20260927 22:51:30.814806 3397 leader_election.cc:288] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:30.814924 3403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:30.817169 3397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:30.819355 3403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [term 1 LEADER]: Becoming Leader. State: Replica: 9107fad6107e4c2b9eb65bdde2941fd5, State: Running, Role: LEADER
I20260927 22:51:30.820101 3403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9107fad6107e4c2b9eb65bdde2941fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } }
I20260927 22:51:30.823112 3405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9107fad6107e4c2b9eb65bdde2941fd5. Latest consensus state: current_term: 1 leader_uuid: "9107fad6107e4c2b9eb65bdde2941fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9107fad6107e4c2b9eb65bdde2941fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } } }
I20260927 22:51:30.823246 3404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9107fad6107e4c2b9eb65bdde2941fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9107fad6107e4c2b9eb65bdde2941fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35755 } } }
I20260927 22:51:30.823787 3405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:30.823828 3404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:30.834887 3417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:30.839813 3417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:30.848541 3417 catalog_manager.cc:1347] Generated new cluster ID: 75c8a1468f664214969d4365afe1deb6
I20260927 22:51:30.848646 3417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:30.850039 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:30.838602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.0.97.62:35755)
0927 22:51:30.838784 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:30.838792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:30.842280 (+ 3488us) server_negotiation.cc:407] Connection header received
0927 22:51:30.842498 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:30.842503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:30.842593 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:30.842728 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:30.843761 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:30.844946 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:30.846639 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:30.847068 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:30.847210 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:30.848267 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:30.848273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:30.848278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:30.849126 (+ 848us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:30.849176 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:30.849209 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:30.849319 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:30.849553 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 22:51:30.849666 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:51:30.864876 3417 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:30.866091 3417 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:30.873212 3417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9107fad6107e4c2b9eb65bdde2941fd5: Generated new TSK 0
I20260927 22:51:30.873991 3417 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:51:30.884516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3330
I20260927 22:51:30.885712 3392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:30.971637 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 3330
2026-09-27T22:51:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1841 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:31 dist-test-slave-df53 krb5kdc[3431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:31 dist-test-slave-df53 krb5kdc[3431](info): set up 2 sockets
Sep 27 22:51:31 dist-test-slave-df53 krb5kdc[3431](info): commencing operation
krb5kdc: starting...
W20260927 22:51:32.029919 388 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:32 dist-test-slave-df53 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:32.208479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45393
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:32.530268 3444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:32.530527 3444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:32.530702 3444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:32.537698 3444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:32.537806 3444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:32.537889 3444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:32.537988 3444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:32.538066 3444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:32.548296 3444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:32.549525 3444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:32.550870 3444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:32.562409 3450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:32.562415 3452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:32.562882 3449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:32.563313 3444 server_base.cc:1034] running on GCE node
I20260927 22:51:32.564055 3444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:32.565368 3444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:32.566596 3444 hybrid_clock.cc:648] HybridClock initialized: now 1790549492566537 us; error 85 us; skew 500 ppm
I20260927 22:51:32.566933 3444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:32 dist-test-slave-df53 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:32.569839 3444 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:32.571302 3444 webserver.cc:460] Webserver started at http://127.0.97.62:34377/ using document root <none> and password file <none>
I20260927 22:51:32.571931 3444 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:32.572068 3444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:32.572360 3444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:32.575194 3444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "aee2fcd4147442cf90b0d1766646d6eb"
format_stamp: "Formatted at 2026-09-27 22:51:32 on dist-test-slave-df53"
server_key: "37ef91e31f06a43b6ed8a0cfbfbccd8c"
server_key_iv: "74ee7262c2388e948b6b7c782081005f"
server_key_version: "encryptionkey@0"
I20260927 22:51:32.575905 3444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "aee2fcd4147442cf90b0d1766646d6eb"
format_stamp: "Formatted at 2026-09-27 22:51:32 on dist-test-slave-df53"
server_key: "37ef91e31f06a43b6ed8a0cfbfbccd8c"
server_key_iv: "74ee7262c2388e948b6b7c782081005f"
server_key_version: "encryptionkey@0"
I20260927 22:51:32.580650 3444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:51:32.583783 3459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:32.584998 3444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 22:51:32.585170 3444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "aee2fcd4147442cf90b0d1766646d6eb"
format_stamp: "Formatted at 2026-09-27 22:51:32 on dist-test-slave-df53"
server_key: "37ef91e31f06a43b6ed8a0cfbfbccd8c"
server_key_iv: "74ee7262c2388e948b6b7c782081005f"
server_key_version: "encryptionkey@0"
I20260927 22:51:32.585363 3444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:32.603945 3444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:32.607714 3444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:32.607987 3444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:32.626163 3444 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45393
I20260927 22:51:32.626180 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45393 every 8 connection(s)
I20260927 22:51:32.627400 3444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:32.630826 3511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:32.634055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 3444
I20260927 22:51:32.634246 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:32.634882 388 external_mini_cluster.cc:1428] Setting key 1dc5bbc9352c8e1144f28ae59596e7a6
I20260927 22:51:32.643370 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb: Bootstrap starting.
I20260927 22:51:32.646721 3511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:32.647639 3511 log.cc:824] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:51:32 dist-test-slave-df53 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:32.650516 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb: No bootstrap required, opened a new log
I20260927 22:51:32.657858 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:32.636694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.0.97.62:45393)
0927 22:51:32.637472 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:32.637486 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:32.637535 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:51:32.639323 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:32.639350 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:32.639833 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:32.640255 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:32.641418 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:32.643293 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:32.644507 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:32.645554 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:32.645768 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:32.649617 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:32.649645 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:32.649671 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:32.649717 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:32.653572 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:32.654181 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:32.654189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:32.654196 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:32.654382 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:32.654820 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:32.654826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:32.654831 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:32.655468 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:32.655765 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:32.656191 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:51:32.656582 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":326,"threads_started":1}
I20260927 22:51:32.660320 3511 raft_consensus.cc:348] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee2fcd4147442cf90b0d1766646d6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } }
I20260927 22:51:32.660670 3511 raft_consensus.cc:374] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:32.660742 3511 raft_consensus.cc:729] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee2fcd4147442cf90b0d1766646d6eb, State: Initialized, Role: FOLLOWER
I20260927 22:51:32.661175 3511 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee2fcd4147442cf90b0d1766646d6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } }
I20260927 22:51:32.661351 3511 raft_consensus.cc:388] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:32.661440 3511 raft_consensus.cc:482] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:32.661589 3511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:32.663837 3511 raft_consensus.cc:504] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee2fcd4147442cf90b0d1766646d6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } }
I20260927 22:51:32.664220 3511 leader_election.cc:302] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aee2fcd4147442cf90b0d1766646d6eb; no voters:
I20260927 22:51:32.664868 3511 leader_election.cc:288] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:32.664913 3516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:32.666136 3516 raft_consensus.cc:686] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [term 1 LEADER]: Becoming Leader. State: Replica: aee2fcd4147442cf90b0d1766646d6eb, State: Running, Role: LEADER
I20260927 22:51:32.666659 3516 consensus_queue.cc:237] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee2fcd4147442cf90b0d1766646d6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } }
I20260927 22:51:32.666849 3511 sys_catalog.cc:564] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:32.669413 3517 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aee2fcd4147442cf90b0d1766646d6eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee2fcd4147442cf90b0d1766646d6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } }
I20260927 22:51:32.669807 3517 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:32.670939 3518 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader aee2fcd4147442cf90b0d1766646d6eb. Latest consensus state: current_term: 1 leader_uuid: "aee2fcd4147442cf90b0d1766646d6eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee2fcd4147442cf90b0d1766646d6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45393 } } }
I20260927 22:51:32.671321 3518 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:32.673115 3523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:32.676964 3523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:32.685467 3523 catalog_manager.cc:1347] Generated new cluster ID: f965aa15e97b4c76ae9454400f4bbd3e
I20260927 22:51:32.685611 3523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:32.713217 3523 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:32.714772 3523 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:32.729075 3523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aee2fcd4147442cf90b0d1766646d6eb: Generated new TSK 0
I20260927 22:51:32.730034 3523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:32.778597 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45393
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:33.091967 3537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:33.092239 3537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:33.092422 3537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:33.099594 3537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:33.099691 3537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:33.099874 3537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:33.109256 3537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:33.110505 3537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:33.111781 3537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:33.124907 3543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:33.125007 3542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:33.124991 3545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:33.126783 3537 server_base.cc:1034] running on GCE node
I20260927 22:51:33.127346 3537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:33.128226 3537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:33.129443 3537 hybrid_clock.cc:648] HybridClock initialized: now 1790549493129395 us; error 63 us; skew 500 ppm
I20260927 22:51:33.129803 3537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:33 dist-test-slave-df53 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:33.133224 3537 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:33.134593 3537 webserver.cc:460] Webserver started at http://127.0.97.1:36441/ using document root <none> and password file <none>
I20260927 22:51:33.135233 3537 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:33.135329 3537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:33.135558 3537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:33.138283 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "42d7dca5fc9b47d8bcfb78264101f68e"
format_stamp: "Formatted at 2026-09-27 22:51:33 on dist-test-slave-df53"
server_key: "a4f42b0bcc7573cc26194bc2e896c824"
server_key_iv: "0a6c74709ee2ef44d982938eba25cf87"
server_key_version: "encryptionkey@0"
I20260927 22:51:33.138983 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "42d7dca5fc9b47d8bcfb78264101f68e"
format_stamp: "Formatted at 2026-09-27 22:51:33 on dist-test-slave-df53"
server_key: "a4f42b0bcc7573cc26194bc2e896c824"
server_key_iv: "0a6c74709ee2ef44d982938eba25cf87"
server_key_version: "encryptionkey@0"
I20260927 22:51:33.143733 3537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:33.146898 3552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:33.148121 3537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:51:33.148288 3537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "42d7dca5fc9b47d8bcfb78264101f68e"
format_stamp: "Formatted at 2026-09-27 22:51:33 on dist-test-slave-df53"
server_key: "a4f42b0bcc7573cc26194bc2e896c824"
server_key_iv: "0a6c74709ee2ef44d982938eba25cf87"
server_key_version: "encryptionkey@0"
I20260927 22:51:33.148471 3537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:33.190879 3537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:33.195176 3537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:33.195468 3537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:33.196367 3537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:33.197685 3537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:33.197757 3537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:33.197916 3537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:33.197976 3537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:33.230357 3537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44855
I20260927 22:51:33.230446 3664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44855 every 8 connection(s)
I20260927 22:51:33.231638 3537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:33.235270 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 3537
I20260927 22:51:33.235459 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:33.236142 388 external_mini_cluster.cc:1428] Setting key 8ede0121e65f59e60c3361e8c2bce20e
Sep 27 22:51:33 dist-test-slave-df53 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:33.252970 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:33.234167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40977 (local address 127.0.97.62:45393)
0927 22:51:33.234573 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:33.234577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:33.235588 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 22:51:33.237460 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:33.237465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:33.237543 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:33.237687 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:33.239610 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:33.240322 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:33.242799 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:33.243186 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:33.243337 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:33.247135 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:33.247140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:33.247143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:33.247171 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:33.249636 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:33.250551 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:33.250557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:33.250561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:33.250700 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:33.251200 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:33.251203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:33.251205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:33.251443 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:33.251553 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:33.252454 (+ 901us) server_negotiation.cc:299] Negotiation successful
0927 22:51:33.252615 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":193,"threads_started":1}
I20260927 22:51:33.254318 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:33.234695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45393 (local address 127.0.97.1:40977)
0927 22:51:33.235427 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:33.235479 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:33.237213 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:33.237902 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:33.237909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:33.238615 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:33.239400 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:33.239437 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:33.240471 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:33.240479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:33.242637 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:33.242654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:33.243407 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:33.243412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:33.243594 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:33.244257 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:33.244301 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:33.246927 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:33.249775 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:33.249787 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:33.249803 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:33.250399 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:33.250818 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:33.250825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:33.250830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:33.251069 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:33.251576 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:33.251587 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:33.252016 (+ 429us) client_negotiation.cc:769] Sending connection context
0927 22:51:33.252360 (+ 344us) client_negotiation.cc:240] Negotiation successful
0927 22:51:33.252915 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":305,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:51:33.255913 3665 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45393
I20260927 22:51:33.256313 3665 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:33.257063 3665 heartbeater.cc:511] Master 127.0.97.62:45393 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:51:33.259267 3476 ts_manager.cc:194] Registered new tserver with Master: 42d7dca5fc9b47d8bcfb78264101f68e (127.0.97.1:44855)
I20260927 22:51:33.260660 3476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:33.278970 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45393
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:33.589677 3671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:33.589982 3671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:33.590178 3671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:33.597177 3671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:33.597292 3671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:33.597456 3671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:33.606971 3671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:33.608188 3671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:33.609400 3671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:33.622138 3679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:33.622218 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:33.622241 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:33.622798 3671 server_base.cc:1034] running on GCE node
I20260927 22:51:33.623397 3671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:33.624318 3671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:33.625582 3671 hybrid_clock.cc:648] HybridClock initialized: now 1790549493625538 us; error 48 us; skew 500 ppm
I20260927 22:51:33.625928 3671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:33 dist-test-slave-df53 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:33.628942 3671 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:33.630393 3671 webserver.cc:460] Webserver started at http://127.0.97.2:43073/ using document root <none> and password file <none>
I20260927 22:51:33.631049 3671 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:33.631171 3671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:33.631465 3671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:33.634300 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "02f03bc7a7744ac799f941165f8ca607"
format_stamp: "Formatted at 2026-09-27 22:51:33 on dist-test-slave-df53"
server_key: "c78af75932a918d850b2357575dcb463"
server_key_iv: "2cab6899bbd9a4df30b4b5306067840c"
server_key_version: "encryptionkey@0"
I20260927 22:51:33.635058 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "02f03bc7a7744ac799f941165f8ca607"
format_stamp: "Formatted at 2026-09-27 22:51:33 on dist-test-slave-df53"
server_key: "c78af75932a918d850b2357575dcb463"
server_key_iv: "2cab6899bbd9a4df30b4b5306067840c"
server_key_version: "encryptionkey@0"
I20260927 22:51:33.639966 3671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:51:33.643277 3686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:33.644543 3671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:33.644714 3671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "02f03bc7a7744ac799f941165f8ca607"
format_stamp: "Formatted at 2026-09-27 22:51:33 on dist-test-slave-df53"
server_key: "c78af75932a918d850b2357575dcb463"
server_key_iv: "2cab6899bbd9a4df30b4b5306067840c"
server_key_version: "encryptionkey@0"
I20260927 22:51:33.644894 3671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:33.655285 3671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:33.659147 3671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:33.659420 3671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:33.660246 3671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:33.661428 3671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:33.661496 3671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:33.661685 3671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:33.661737 3671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:33.694077 3671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36307
I20260927 22:51:33.694121 3798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36307 every 8 connection(s)
I20260927 22:51:33.695293 3671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:33.704905 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 3671
I20260927 22:51:33.705082 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:33.705540 388 external_mini_cluster.cc:1428] Setting key eda0dd73188332f27a981f5f5ff69e49
Sep 27 22:51:33 dist-test-slave-df53 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:33.715107 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:33.697768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47061 (local address 127.0.97.62:45393)
0927 22:51:33.697944 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:33.697949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:33.699060 (+ 1111us) server_negotiation.cc:407] Connection header received
0927 22:51:33.700757 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:33.700763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:33.700849 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:33.700962 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:33.702428 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:33.703142 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:33.704664 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:33.705121 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:33.705260 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:33.708647 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:33.708656 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:33.708664 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:33.708710 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:33.712226 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:33.712929 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:33.712941 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:33.712943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:33.713059 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:33.713466 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:33.713471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:33.713475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:33.713781 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:33.713938 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:33.714567 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 22:51:33.714782 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:51:33.716020 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:33.698256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45393 (local address 127.0.97.2:47061)
0927 22:51:33.698910 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:33.698950 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:33.700504 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:33.701173 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:33.701180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:33.701707 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:33.702274 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:33.702290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:33.703296 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:33.703300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:33.704482 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:33.704496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:33.705343 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:33.705349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:33.705570 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:33.706316 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:33.706338 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:33.708432 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:33.712353 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:33.712360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:33.712371 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:33.712802 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:33.713168 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:33.713174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:33.713177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:33.713351 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:33.713912 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:33.713919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:33.714225 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 22:51:33.714473 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 22:51:33.714976 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
I20260927 22:51:33.717492 3799 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45393
I20260927 22:51:33.717829 3799 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:33.718642 3799 heartbeater.cc:511] Master 127.0.97.62:45393 requested a full tablet report, sending...
I20260927 22:51:33.720227 3476 ts_manager.cc:194] Registered new tserver with Master: 02f03bc7a7744ac799f941165f8ca607 (127.0.97.2:36307)
I20260927 22:51:33.721068 3476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47061
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:33.749836 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45393
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:34.054673 3804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:34.054904 3804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:34.055045 3804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:34.061833 3804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:34.061918 3804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:34.062088 3804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:34.071426 3804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:34.072602 3804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:34.073830 3804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:34.085994 3809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:34.086021 3810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:34.086315 3812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:34.086916 3804 server_base.cc:1034] running on GCE node
I20260927 22:51:34.087448 3804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:34.088325 3804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:34.089552 3804 hybrid_clock.cc:648] HybridClock initialized: now 1790549494089508 us; error 40 us; skew 500 ppm
I20260927 22:51:34.089991 3804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:34 dist-test-slave-df53 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:34.099375 3804 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:34.100858 3804 webserver.cc:460] Webserver started at http://127.0.97.3:33113/ using document root <none> and password file <none>
I20260927 22:51:34.101521 3804 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:34.101686 3804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:34.101981 3804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:34.104722 3804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "cb7c09d5050f4b11a07b066c965e125a"
format_stamp: "Formatted at 2026-09-27 22:51:34 on dist-test-slave-df53"
server_key: "2628409fb4de24adf2626299a7c0a848"
server_key_iv: "b6d62a91d544b9542fc122a1a7f25c24"
server_key_version: "encryptionkey@0"
I20260927 22:51:34.105469 3804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cb7c09d5050f4b11a07b066c965e125a"
format_stamp: "Formatted at 2026-09-27 22:51:34 on dist-test-slave-df53"
server_key: "2628409fb4de24adf2626299a7c0a848"
server_key_iv: "b6d62a91d544b9542fc122a1a7f25c24"
server_key_version: "encryptionkey@0"
I20260927 22:51:34.110167 3804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:51:34.113366 3819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:34.114993 3804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:34.115165 3804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "cb7c09d5050f4b11a07b066c965e125a"
format_stamp: "Formatted at 2026-09-27 22:51:34 on dist-test-slave-df53"
server_key: "2628409fb4de24adf2626299a7c0a848"
server_key_iv: "b6d62a91d544b9542fc122a1a7f25c24"
server_key_version: "encryptionkey@0"
I20260927 22:51:34.115345 3804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:34.147568 3804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:34.151397 3804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:34.151705 3804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:34.152637 3804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:34.153975 3804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:34.154071 3804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:34.154217 3804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:34.154286 3804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:34.186213 3804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34099
I20260927 22:51:34.186297 3931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34099 every 8 connection(s)
I20260927 22:51:34.187422 3804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:34.196993 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 3804
I20260927 22:51:34.197183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:34.197692 388 external_mini_cluster.cc:1428] Setting key 0c026ab59ef40e87d84848b38dea8262
Sep 27 22:51:34 dist-test-slave-df53 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:34.206811 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:34.190333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58863 (local address 127.0.97.62:45393)
0927 22:51:34.190520 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:34.190524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:34.191726 (+ 1202us) server_negotiation.cc:407] Connection header received
0927 22:51:34.193465 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:34.193470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:34.193556 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:34.193691 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:34.195063 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.195816 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.197463 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.197973 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.198104 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:34.201761 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:34.201771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:34.201774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:34.201803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:34.204045 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:34.204730 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:34.204737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:34.204743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:34.204858 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:34.205253 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:34.205260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:34.205263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:34.205501 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:34.205671 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:34.206289 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 22:51:34.206486 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:51:34.207657 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:34.190859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45393 (local address 127.0.97.3:58863)
0927 22:51:34.191547 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:34.191583 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:34.193219 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:34.193918 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:34.193926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:34.194385 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:34.194901 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:34.194917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.195968 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.195972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:34.197332 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:34.197343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.198192 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.198201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:34.198484 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:34.199217 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:34.199245 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:34.201537 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:34.204210 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:34.204218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:34.204233 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:34.204613 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:34.204963 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:34.204969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:34.204973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:34.205149 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:34.205649 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:34.205657 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:34.205942 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 22:51:34.206190 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 22:51:34.206656 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":315,"threads_started":1}
I20260927 22:51:34.208945 3932 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45393
I20260927 22:51:34.209218 3932 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:34.209923 3932 heartbeater.cc:511] Master 127.0.97.62:45393 requested a full tablet report, sending...
I20260927 22:51:34.211197 3476 ts_manager.cc:194] Registered new tserver with Master: cb7c09d5050f4b11a07b066c965e125a (127.0.97.3:34099)
I20260927 22:51:34.211987 3476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58863
I20260927 22:51:34.212608 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:51:34 dist-test-slave-df53 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549494, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 22:51:34 dist-test-slave-df53 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549494, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:34.256681 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:34.243411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.0.97.62:45393)
0927 22:51:34.243613 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:34.243617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:34.244125 (+ 508us) server_negotiation.cc:407] Connection header received
0927 22:51:34.244255 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:34.244258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:34.244314 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:34.244401 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:34.245708 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.246391 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.247543 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.247936 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.248032 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:34.251746 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:34.251750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:34.251754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:34.251782 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:34.254658 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:34.255194 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:34.255199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:34.255201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:34.255317 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:34.255647 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:34.255650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:34.255652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:34.255911 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:34.256077 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:34.256316 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 22:51:34.256455 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 22:51:34.257166 3476 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45092
I20260927 22:51:34.258849 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3537
I20260927 22:51:34.263235 3665 heartbeater.cc:503] Master 127.0.97.62:45393 was elected leader, sending a full tablet report...
Sep 27 22:51:34 dist-test-slave-df53 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549494, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:34.274688 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:34.259917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.0.97.1:44855)
0927 22:51:34.260460 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:34.260466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:34.260487 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:51:34.260531 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:34.260536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:34.260718 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:34.260827 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:34.261998 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.262956 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.264263 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.264984 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.265079 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:34.269076 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:34.269085 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:34.269098 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:34.269141 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:34.272165 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:34.272802 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:34.272809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:34.272812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:34.272941 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:34.273291 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:34.273294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:34.273296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:34.273676 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:34.273915 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:34.274208 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 22:51:34.274389 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":304,"threads_started":1}
W20260927 22:51:34.275838 3660 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43896
W20260927 22:51:34.276855 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3537: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:51:34.277053 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 3537
I20260927 22:51:34.291937 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3671
Sep 27 22:51:34 dist-test-slave-df53 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549494, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:34.306974 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:34.292708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.0.97.2:36307)
0927 22:51:34.293254 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:34.293261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:34.293285 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:51:34.293341 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:34.293346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:34.293538 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:34.293727 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:34.294819 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.295813 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.296957 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.297477 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.297573 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:34.301421 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:34.301431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:34.301445 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:34.301472 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:34.304322 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:34.304929 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:34.304935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:34.304937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:34.305052 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:34.305437 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:34.305439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:34.305441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:34.305957 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:34.306182 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:34.306417 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 22:51:34.306596 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":273,"threads_started":1}
W20260927 22:51:34.308172 3794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52076
W20260927 22:51:34.309288 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3671: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:51:34.309441 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 3671
I20260927 22:51:34.324781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3804
Sep 27 22:51:34 dist-test-slave-df53 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549494, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:51:34.340340 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:34.325729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.0.97.3:34099)
0927 22:51:34.326353 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:34.326360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:34.326387 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:51:34.326442 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:34.326448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:34.326638 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:34.326810 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:34.328054 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.329043 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.330284 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:34.330823 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:34.330953 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:34.334982 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:34.334992 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:34.335005 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:34.335034 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:34.337549 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:34.338389 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:34.338397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:34.338403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:34.338521 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:34.338876 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:34.338881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:34.338884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:34.339199 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:34.339504 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:34.339769 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 22:51:34.340019 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":329,"threads_started":1}
W20260927 22:51:34.341508 3927 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58672
W20260927 22:51:34.342584 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3804: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:51:34.342769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 3804
I20260927 22:51:34.358992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3444
I20260927 22:51:34.360028 3506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:34.459357 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 3444
2026-09-27T22:51:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3493 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:34 dist-test-slave-df53 krb5kdc[3949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:34 dist-test-slave-df53 krb5kdc[3949](info): set up 2 sockets
Sep 27 22:51:34 dist-test-slave-df53 krb5kdc[3949](info): commencing operation
krb5kdc: starting...
W20260927 22:51:35.526042 388 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:35 dist-test-slave-df53 krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:35.709414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42101
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:36.012344 3962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:36.012591 3962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:36.012742 3962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:36.019567 3962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:36.019655 3962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:36.019752 3962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:36.019845 3962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:36.019922 3962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:36.029106 3962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:36.030267 3962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:36.031486 3962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:36.042820 3968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:36.042922 3967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:36.042891 3970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:36.043741 3962 server_base.cc:1034] running on GCE node
I20260927 22:51:36.044409 3962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:36.045656 3962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:36.046882 3962 hybrid_clock.cc:648] HybridClock initialized: now 1790549496046847 us; error 26 us; skew 500 ppm
I20260927 22:51:36.047171 3962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:36 dist-test-slave-df53 krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:36.050036 3962 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:36.051405 3962 webserver.cc:460] Webserver started at http://127.0.97.62:42391/ using document root <none> and password file <none>
I20260927 22:51:36.052050 3962 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:36.052186 3962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:36.052505 3962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:36.055238 3962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "37fba5421d1f44f2a2a63f73c0a41ac7"
format_stamp: "Formatted at 2026-09-27 22:51:36 on dist-test-slave-df53"
server_key: "af11ac0cee415f1f7bb790d93a4fd73c"
server_key_iv: "630eed26daa00da67c37e4a3045111f8"
server_key_version: "encryptionkey@0"
I20260927 22:51:36.056003 3962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "37fba5421d1f44f2a2a63f73c0a41ac7"
format_stamp: "Formatted at 2026-09-27 22:51:36 on dist-test-slave-df53"
server_key: "af11ac0cee415f1f7bb790d93a4fd73c"
server_key_iv: "630eed26daa00da67c37e4a3045111f8"
server_key_version: "encryptionkey@0"
I20260927 22:51:36.060757 3962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:51:36.064165 3977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:36.065418 3962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:51:36.065639 3962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "37fba5421d1f44f2a2a63f73c0a41ac7"
format_stamp: "Formatted at 2026-09-27 22:51:36 on dist-test-slave-df53"
server_key: "af11ac0cee415f1f7bb790d93a4fd73c"
server_key_iv: "630eed26daa00da67c37e4a3045111f8"
server_key_version: "encryptionkey@0"
I20260927 22:51:36.065812 3962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:36.088454 3962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:36.092363 3962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:36.092656 3962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:36.110725 3962 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42101
I20260927 22:51:36.110775 4028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42101 every 8 connection(s)
I20260927 22:51:36.112020 3962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:36.114614 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 3962
I20260927 22:51:36.114848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:36.115595 388 external_mini_cluster.cc:1428] Setting key 853b8626c46b7535519dbaf31065fd16
I20260927 22:51:36.116257 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:36.130100 4029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7: Bootstrap starting.
Sep 27 22:51:36 dist-test-slave-df53 krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:36.133816 4029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:36.135089 4029 log.cc:824] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:36.138235 4029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7: No bootstrap required, opened a new log
I20260927 22:51:36.139513 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:36.117678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.0.97.62:42101)
0927 22:51:36.118466 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:36.118476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:36.118515 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:51:36.120628 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:36.120648 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:36.121016 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:36.121347 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:36.123087 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:36.124316 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:36.126080 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:36.126763 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:36.126906 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:36.131453 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:36.131464 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:36.131487 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:36.131540 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:36.135330 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:36.136123 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:36.136129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:36.136134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:36.136297 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:36.136699 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:36.136703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:36.136706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:36.137275 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:36.137570 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:36.137870 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:51:36.138270 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":394,"threads_started":1}
I20260927 22:51:36.146373 4029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fba5421d1f44f2a2a63f73c0a41ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } }
I20260927 22:51:36.146651 4029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:36.146718 4029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fba5421d1f44f2a2a63f73c0a41ac7, State: Initialized, Role: FOLLOWER
I20260927 22:51:36.147202 4029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fba5421d1f44f2a2a63f73c0a41ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } }
I20260927 22:51:36.147384 4029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:36.147475 4029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:36.147605 4029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:36.149808 4029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fba5421d1f44f2a2a63f73c0a41ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } }
I20260927 22:51:36.150203 4029 leader_election.cc:302] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37fba5421d1f44f2a2a63f73c0a41ac7; no voters:
I20260927 22:51:36.150684 4029 leader_election.cc:288] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:36.150818 4034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:36.151490 4034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [term 1 LEADER]: Becoming Leader. State: Replica: 37fba5421d1f44f2a2a63f73c0a41ac7, State: Running, Role: LEADER
I20260927 22:51:36.152019 4034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fba5421d1f44f2a2a63f73c0a41ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } }
I20260927 22:51:36.152592 4029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:36.155081 4036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37fba5421d1f44f2a2a63f73c0a41ac7. Latest consensus state: current_term: 1 leader_uuid: "37fba5421d1f44f2a2a63f73c0a41ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fba5421d1f44f2a2a63f73c0a41ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } } }
I20260927 22:51:36.155496 4036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:36.156760 4042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:36.156733 4035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37fba5421d1f44f2a2a63f73c0a41ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fba5421d1f44f2a2a63f73c0a41ac7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } } }
I20260927 22:51:36.157133 4035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:36.160861 4042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:36.169428 4042 catalog_manager.cc:1347] Generated new cluster ID: 9742c8f9c74747019324ed95be71bda3
I20260927 22:51:36.169543 4042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:36.179420 4042 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:36.180630 4042 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:36.186254 4042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37fba5421d1f44f2a2a63f73c0a41ac7: Generated new TSK 0
I20260927 22:51:36.187088 4042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:36.236322 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42101
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:36.546007 4055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:36.546275 4055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:36.546448 4055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:36.553725 4055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:36.553845 4055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:36.554028 4055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:36.563740 4055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:36.565035 4055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:36.566252 4055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:36.579344 4060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:36.579305 4061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:36.579398 4063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:36.580370 4055 server_base.cc:1034] running on GCE node
I20260927 22:51:36.580919 4055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:36.581830 4055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:36.583022 4055 hybrid_clock.cc:648] HybridClock initialized: now 1790549496582964 us; error 48 us; skew 500 ppm
I20260927 22:51:36.583319 4055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:36 dist-test-slave-df53 krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:36.586750 4055 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:36.588030 4055 webserver.cc:460] Webserver started at http://127.0.97.1:36395/ using document root <none> and password file <none>
I20260927 22:51:36.588665 4055 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:36.588791 4055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:36.589087 4055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:36.591820 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "f8d9dad96746465286747d36e51269d6"
format_stamp: "Formatted at 2026-09-27 22:51:36 on dist-test-slave-df53"
server_key: "b14eb17bf6c398deda127478a5fccb86"
server_key_iv: "642b917ab7da8f9bb4b5678493c90c0c"
server_key_version: "encryptionkey@0"
I20260927 22:51:36.592654 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f8d9dad96746465286747d36e51269d6"
format_stamp: "Formatted at 2026-09-27 22:51:36 on dist-test-slave-df53"
server_key: "b14eb17bf6c398deda127478a5fccb86"
server_key_iv: "642b917ab7da8f9bb4b5678493c90c0c"
server_key_version: "encryptionkey@0"
I20260927 22:51:36.597302 4055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260927 22:51:36.600421 4070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:36.601698 4055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 22:51:36.601847 4055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "f8d9dad96746465286747d36e51269d6"
format_stamp: "Formatted at 2026-09-27 22:51:36 on dist-test-slave-df53"
server_key: "b14eb17bf6c398deda127478a5fccb86"
server_key_iv: "642b917ab7da8f9bb4b5678493c90c0c"
server_key_version: "encryptionkey@0"
I20260927 22:51:36.602048 4055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:36.625291 4055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:36.628877 4055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:36.629153 4055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:36.630028 4055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:36.631273 4055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:36.631367 4055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:36.631510 4055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:36.631580 4055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:36.663399 4055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40507
I20260927 22:51:36.663455 4182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40507 every 8 connection(s)
I20260927 22:51:36.664671 4055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:36.672040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 4055
I20260927 22:51:36.672271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:36.672879 388 external_mini_cluster.cc:1428] Setting key 9b649b51dce9b2f4f0385e528fd6e1ac
Sep 27 22:51:36 dist-test-slave-df53 krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:36.685133 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:36.667118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53391 (local address 127.0.97.62:42101)
0927 22:51:36.667532 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:36.667538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:36.668595 (+ 1057us) server_negotiation.cc:407] Connection header received
0927 22:51:36.670490 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:36.670494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:36.670554 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:36.670654 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:36.672256 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:36.673022 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:36.674982 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:36.675347 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:36.675434 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:36.679234 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:36.679243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:36.679250 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:36.679296 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:36.681968 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:36.682848 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:36.682852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:36.682854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:36.682960 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:36.683486 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:36.683490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:36.683492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:36.683785 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:36.683898 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:36.684721 (+ 823us) server_negotiation.cc:299] Negotiation successful
0927 22:51:36.684871 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":198,"threads_started":1}
I20260927 22:51:36.686578 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:36.667709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42101 (local address 127.0.97.1:53391)
0927 22:51:36.668384 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:36.668429 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:36.670265 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:36.670834 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:36.670865 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:36.671328 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:36.672069 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:36.672093 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:36.673163 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:36.673169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:36.674837 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:36.674849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:36.675558 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:36.675566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:36.675879 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:36.676558 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:36.676589 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:36.679042 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:36.682090 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:36.682100 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:36.682124 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:36.682724 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:36.683069 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:36.683076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:36.683082 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:36.683330 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:36.683920 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:36.683931 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:36.684338 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 22:51:36.684633 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 22:51:36.685170 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":269,"threads_started":1}
I20260927 22:51:36.688169 4183 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42101
I20260927 22:51:36.688479 4183 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:36.689175 4183 heartbeater.cc:511] Master 127.0.97.62:42101 requested a full tablet report, sending...
I20260927 22:51:36.691138 3994 ts_manager.cc:194] Registered new tserver with Master: f8d9dad96746465286747d36e51269d6 (127.0.97.1:40507)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:51:36.692526 3994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:36.716470 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42101
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:37.028463 4189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:37.028717 4189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:37.028807 4189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:37.035723 4189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:37.035812 4189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:37.035914 4189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:37.044857 4189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:37.045946 4189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:37.047060 4189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:37.059525 4197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:37.059572 4194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:37.059484 4195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:37.063601 4189 server_base.cc:1034] running on GCE node
I20260927 22:51:37.064254 4189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:37.065143 4189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:37.066356 4189 hybrid_clock.cc:648] HybridClock initialized: now 1790549497066326 us; error 48 us; skew 500 ppm
I20260927 22:51:37.066658 4189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:37 dist-test-slave-df53 krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:37.069559 4189 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:37.071004 4189 webserver.cc:460] Webserver started at http://127.0.97.2:34341/ using document root <none> and password file <none>
I20260927 22:51:37.071611 4189 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:37.071705 4189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:37.071933 4189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:37.074590 4189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "3b79683eadb34228ac641992ac98e469"
format_stamp: "Formatted at 2026-09-27 22:51:37 on dist-test-slave-df53"
server_key: "45d1dd2f8932decf4526b86f91f97e0d"
server_key_iv: "bc8386b7bd8193d5fbe6c5faf18ee32c"
server_key_version: "encryptionkey@0"
I20260927 22:51:37.075268 4189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3b79683eadb34228ac641992ac98e469"
format_stamp: "Formatted at 2026-09-27 22:51:37 on dist-test-slave-df53"
server_key: "45d1dd2f8932decf4526b86f91f97e0d"
server_key_iv: "bc8386b7bd8193d5fbe6c5faf18ee32c"
server_key_version: "encryptionkey@0"
I20260927 22:51:37.079931 4189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:37.082981 4204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:37.084233 4189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:51:37.084357 4189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "3b79683eadb34228ac641992ac98e469"
format_stamp: "Formatted at 2026-09-27 22:51:37 on dist-test-slave-df53"
server_key: "45d1dd2f8932decf4526b86f91f97e0d"
server_key_iv: "bc8386b7bd8193d5fbe6c5faf18ee32c"
server_key_version: "encryptionkey@0"
I20260927 22:51:37.084519 4189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:37.107329 4189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:37.111129 4189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:37.111405 4189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:37.112278 4189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:37.113497 4189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:37.113597 4189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:37.113773 4189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:37.113853 4189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:37.146462 4189 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33447
I20260927 22:51:37.146549 4316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33447 every 8 connection(s)
I20260927 22:51:37.147698 4189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:37.151906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 4189
I20260927 22:51:37.152122 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:37.152690 388 external_mini_cluster.cc:1428] Setting key 6ffbf705a318f4e56f0c9245bbd35427
Sep 27 22:51:37 dist-test-slave-df53 krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:37.170490 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.150119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37711 (local address 127.0.97.62:42101)
0927 22:51:37.150278 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:37.150283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:37.151416 (+ 1133us) server_negotiation.cc:407] Connection header received
0927 22:51:37.153943 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:37.153948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:37.154007 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:37.154127 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:37.155989 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.156681 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.159130 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.159495 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.159582 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:37.163366 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:37.163374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:37.163381 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:37.163423 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:37.167071 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.167798 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.167807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.167814 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.167971 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.168513 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.168518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.168521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.168922 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:37.169096 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:37.169815 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 22:51:37.170053 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:51:37.171178 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.150577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42101 (local address 127.0.97.2:37711)
0927 22:51:37.151268 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:37.151303 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:37.153569 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:37.154392 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:37.154405 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:37.155052 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:37.155814 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:37.155840 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.156823 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.156831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:37.158989 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:37.159006 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.159719 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.159729 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:37.160072 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:37.160841 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:37.160861 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:37.163152 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:37.167212 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:37.167218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:37.167229 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:37.167663 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:37.168124 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:37.168130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:37.168134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:37.168371 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:37.169071 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:37.169077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:37.169399 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 22:51:37.169687 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 22:51:37.170080 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":274,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:51:37.172593 4317 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42101
I20260927 22:51:37.172887 4317 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:37.173671 4317 heartbeater.cc:511] Master 127.0.97.62:42101 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:51:37.175158 3994 ts_manager.cc:194] Registered new tserver with Master: 3b79683eadb34228ac641992ac98e469 (127.0.97.2:33447)
I20260927 22:51:37.176385 3994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:37.196300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42101
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:37.502486 4322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:37.502785 4322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:37.502954 4322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:37.510071 4322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:37.510200 4322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:37.510377 4322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:37.519467 4322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:37.520668 4322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:37.521907 4322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:37.534053 4328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:37.534170 4327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:37.534185 4330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:37.534734 4322 server_base.cc:1034] running on GCE node
I20260927 22:51:37.535203 4322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:37.536008 4322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:37.537220 4322 hybrid_clock.cc:648] HybridClock initialized: now 1790549497537188 us; error 48 us; skew 500 ppm
I20260927 22:51:37.537505 4322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:37 dist-test-slave-df53 krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:37.540352 4322 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:37.541649 4322 webserver.cc:460] Webserver started at http://127.0.97.3:43179/ using document root <none> and password file <none>
I20260927 22:51:37.542232 4322 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:37.542322 4322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:37.542563 4322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:37.545133 4322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "00e9c0d85a3e41168d76503e23f447f9"
format_stamp: "Formatted at 2026-09-27 22:51:37 on dist-test-slave-df53"
server_key: "60f94b07d941495d83723617a26ea2bb"
server_key_iv: "3c4174d18f8d915b78013ff62c725fc9"
server_key_version: "encryptionkey@0"
I20260927 22:51:37.545886 4322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "00e9c0d85a3e41168d76503e23f447f9"
format_stamp: "Formatted at 2026-09-27 22:51:37 on dist-test-slave-df53"
server_key: "60f94b07d941495d83723617a26ea2bb"
server_key_iv: "3c4174d18f8d915b78013ff62c725fc9"
server_key_version: "encryptionkey@0"
I20260927 22:51:37.550356 4322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:51:37.553396 4337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:37.554701 4322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:51:37.554826 4322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "00e9c0d85a3e41168d76503e23f447f9"
format_stamp: "Formatted at 2026-09-27 22:51:37 on dist-test-slave-df53"
server_key: "60f94b07d941495d83723617a26ea2bb"
server_key_iv: "3c4174d18f8d915b78013ff62c725fc9"
server_key_version: "encryptionkey@0"
I20260927 22:51:37.554978 4322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:37.569269 4322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:37.574546 4322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:37.574853 4322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:37.575929 4322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:37.577764 4322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:37.577833 4322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:37.577914 4322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:37.577957 4322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:37.609936 4322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43173
I20260927 22:51:37.609967 4449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43173 every 8 connection(s)
I20260927 22:51:37.611169 4322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:37.621567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 4322
I20260927 22:51:37.621797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:37.622397 388 external_mini_cluster.cc:1428] Setting key 4ad3612df36b6377a9581c3d88448891
Sep 27 22:51:37 dist-test-slave-df53 krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:37.630123 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.613560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48735 (local address 127.0.97.62:42101)
0927 22:51:37.613797 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:37.613803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:37.614999 (+ 1196us) server_negotiation.cc:407] Connection header received
0927 22:51:37.616658 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:37.616662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:37.616723 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:37.616809 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:37.618267 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.618982 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.620503 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.620886 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.620974 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:37.625110 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:37.625136 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:37.625142 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:37.625183 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:37.627546 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.628186 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.628190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.628193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.628295 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.628714 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.628718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.628721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.628985 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:37.629087 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:37.629745 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 22:51:37.629893 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 22:51:37.630959 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.614078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42101 (local address 127.0.97.3:48735)
0927 22:51:37.614840 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:37.614880 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:37.616414 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:37.617014 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:37.617022 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:37.617532 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:37.618127 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:37.618149 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.619119 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.619122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:37.620370 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:37.620381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.621054 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.621058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:37.621358 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:37.622152 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:37.622181 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:37.624921 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:37.627682 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:37.627688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:37.627699 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:37.628078 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:37.628406 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:37.628410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:37.628413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:37.628596 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:37.629093 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:37.629098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:37.629365 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 22:51:37.629653 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 22:51:37.629989 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":329,"threads_started":1}
I20260927 22:51:37.632247 4450 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42101
I20260927 22:51:37.632535 4450 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:37.633172 4450 heartbeater.cc:511] Master 127.0.97.62:42101 requested a full tablet report, sending...
I20260927 22:51:37.634481 3994 ts_manager.cc:194] Registered new tserver with Master: 00e9c0d85a3e41168d76503e23f447f9 (127.0.97.3:43173)
I20260927 22:51:37.635216 3994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48735
I20260927 22:51:37.637490 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:51:37 dist-test-slave-df53 krb5kdc[3949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:51:37 dist-test-slave-df53 krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:37.682356 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.668095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.0.97.1:40507)
0927 22:51:37.668806 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:37.668815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:37.669046 (+ 231us) server_negotiation.cc:407] Connection header received
0927 22:51:37.669205 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:37.669211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:37.669469 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:37.669650 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:37.670923 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.672106 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.673264 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.673855 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.673969 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:37.677448 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:37.677457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:37.677469 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:37.677507 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:37.679986 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.680549 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.680555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.680557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.680684 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.681016 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.681019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.681021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.681319 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:37.681534 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:37.681846 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 22:51:37.682079 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":331,"threads_started":1}
Sep 27 22:51:37 dist-test-slave-df53 krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:37.695791 4183 heartbeater.cc:503] Master 127.0.97.62:42101 was elected leader, sending a full tablet report...
I20260927 22:51:37.699184 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.686360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.0.97.62:42101)
0927 22:51:37.686558 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:37.686565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:37.686646 (+ 81us) server_negotiation.cc:407] Connection header received
0927 22:51:37.686762 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:37.686765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:37.686822 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:37.686897 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:37.688035 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.688821 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.690044 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.690413 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.690512 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:37.694142 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:37.694147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:37.694150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:37.694179 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:37.696740 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.697458 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.697462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.697464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.697579 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.697948 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.697951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.697953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.698231 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:37.698414 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:37.698706 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 22:51:37.698842 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 22:51:37.699756 3994 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38500
I20260927 22:51:37.700604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 4055
I20260927 22:51:37.712880 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.701457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.0.97.1:40507)
0927 22:51:37.701673 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:37.701681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:37.701861 (+ 180us) server_negotiation.cc:407] Connection header received
0927 22:51:37.702056 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:37.702063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:37.702152 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:37.702264 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:37.703349 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.704013 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.705829 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.706253 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.706356 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:37.708431 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:37.708436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:37.708439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:37.708467 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:37.710974 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.711527 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.711530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.711532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.711643 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.711928 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.711931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.711933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.712152 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:37.712320 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:37.712547 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 22:51:37.712681 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:37.713434 4178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:37.794991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 4055
I20260927 22:51:37.810374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 4189
Sep 27 22:51:37 dist-test-slave-df53 krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:37.825079 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.811173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.0.97.2:33447)
0927 22:51:37.811684 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:37.811691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:37.811716 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:51:37.811775 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:37.811780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:37.811987 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:37.812120 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:37.813327 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.814348 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.815572 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.816079 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.816167 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:37.820177 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:37.820187 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:37.820201 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:37.820238 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:37.822727 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.823328 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.823335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.823337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.823461 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.823793 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.823796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.823798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.824132 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:37.824356 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:37.824606 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 22:51:37.824786 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":261,"threads_started":1}
I20260927 22:51:37.826495 4312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:37.908694 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 4189
I20260927 22:51:37.924353 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 4322
Sep 27 22:51:37 dist-test-slave-df53 krb5kdc[3949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:51:37.939873 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:37.925316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.0.97.3:43173)
0927 22:51:37.926009 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:37.926016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:37.926044 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:51:37.926101 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:37.926106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:37.926308 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:37.926438 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:37.927639 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.928602 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.930167 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:37.930731 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:37.930844 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:37.934986 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:37.934997 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:37.935011 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:37.935039 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:37.937525 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.938205 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.938213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.938215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.938331 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:37.938661 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:37.938664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:37.938666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:37.938975 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:37.939200 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:37.939407 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 22:51:37.939592 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":414,"threads_started":1}
I20260927 22:51:37.941079 4445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:38.022378 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 4322
I20260927 22:51:38.037673 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 3962
I20260927 22:51:38.038807 4024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:38.138087 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 3962
2026-09-27T22:51:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3678 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:38 dist-test-slave-df53 krb5kdc[4469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:38 dist-test-slave-df53 krb5kdc[4469](info): set up 2 sockets
Sep 27 22:51:38 dist-test-slave-df53 krb5kdc[4469](info): commencing operation
krb5kdc: starting...
W20260927 22:51:39.205865 388 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:39 dist-test-slave-df53 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:39.385344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33423
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:39.693749 4482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:39.694054 4482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:39.694208 4482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:39.701129 4482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:39.701262 4482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:39.701367 4482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:39.701478 4482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:39.701563 4482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:39.710928 4482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:39.712088 4482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:39.713346 4482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:39.724233 4490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:39.724309 4487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:39.724284 4488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:39.725255 4482 server_base.cc:1034] running on GCE node
I20260927 22:51:39.726001 4482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:39.727175 4482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:39.728363 4482 hybrid_clock.cc:648] HybridClock initialized: now 1790549499728318 us; error 39 us; skew 500 ppm
I20260927 22:51:39.728691 4482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:39 dist-test-slave-df53 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:39.736886 4482 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:39.738022 4482 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:39.738133 4482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:39.738440 4482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:39.741153 4482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2"
format_stamp: "Formatted at 2026-09-27 22:51:39 on dist-test-slave-df53"
server_key: "65a730fd2c63858478bba913def48fcc"
server_key_iv: "a93d21e13e86c38027142727c87b1aa2"
server_key_version: "encryptionkey@0"
I20260927 22:51:39.741971 4482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2"
format_stamp: "Formatted at 2026-09-27 22:51:39 on dist-test-slave-df53"
server_key: "65a730fd2c63858478bba913def48fcc"
server_key_iv: "a93d21e13e86c38027142727c87b1aa2"
server_key_version: "encryptionkey@0"
I20260927 22:51:39.746582 4482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 22:51:39.749863 4496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:39.751104 4482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:51:39.751274 4482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2"
format_stamp: "Formatted at 2026-09-27 22:51:39 on dist-test-slave-df53"
server_key: "65a730fd2c63858478bba913def48fcc"
server_key_iv: "a93d21e13e86c38027142727c87b1aa2"
server_key_version: "encryptionkey@0"
I20260927 22:51:39.751490 4482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:39.764091 4482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:39.767670 4482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:39.767925 4482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:39.785519 4482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33423
I20260927 22:51:39.785543 4547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33423 every 8 connection(s)
I20260927 22:51:39.786384 4482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:39.789836 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:39.791061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 4482
I20260927 22:51:39.791280 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:39.791885 388 external_mini_cluster.cc:1428] Setting key 4f8d1ad70649afae52918339f4dea5e6
I20260927 22:51:39.802874 4548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2: Bootstrap starting.
Sep 27 22:51:39 dist-test-slave-df53 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:39.806576 4548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:39.807610 4548 log.cc:824] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:39.810402 4548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2: No bootstrap required, opened a new log
I20260927 22:51:39.814429 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:39.793801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.0.97.62:33423)
0927 22:51:39.794578 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:39.794589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:39.794634 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:51:39.796155 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:39.796175 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:39.796531 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:39.796826 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:39.798615 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:39.799945 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:39.801705 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:39.802316 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:39.802439 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:39.807095 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:39.807102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:39.807115 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:39.807153 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:39.810617 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:39.811295 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:39.811301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:39.811306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:39.811457 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:39.811838 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:39.811842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:39.811844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:39.812339 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:39.812668 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:39.813038 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 22:51:39.813406 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":579,"thread_start_us":388,"threads_started":1}
I20260927 22:51:39.817852 4548 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33423 } }
I20260927 22:51:39.818199 4548 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:39.818315 4548 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ceffa4caf64c46ba3e8f5f3ceb51d2, State: Initialized, Role: FOLLOWER
I20260927 22:51:39.818811 4548 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33423 } }
I20260927 22:51:39.819028 4548 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:39.819166 4548 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:39.819327 4548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:39.821508 4548 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33423 } }
I20260927 22:51:39.821993 4548 leader_election.cc:302] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ceffa4caf64c46ba3e8f5f3ceb51d2; no voters:
I20260927 22:51:39.822494 4548 leader_election.cc:288] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:39.822611 4553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:39.823264 4553 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [term 1 LEADER]: Becoming Leader. State: Replica: e0ceffa4caf64c46ba3e8f5f3ceb51d2, State: Running, Role: LEADER
I20260927 22:51:39.823766 4553 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33423 } }
I20260927 22:51:39.824348 4548 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:39.826530 4554 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33423 } } }
I20260927 22:51:39.826848 4554 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:39.826480 4555 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ceffa4caf64c46ba3e8f5f3ceb51d2. Latest consensus state: current_term: 1 leader_uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ceffa4caf64c46ba3e8f5f3ceb51d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33423 } } }
I20260927 22:51:39.827370 4555 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:39.831321 4558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:39.835844 4558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:39.843025 4558 catalog_manager.cc:1347] Generated new cluster ID: ede4bde44f194ffb8ebe157c8f08c685
I20260927 22:51:39.843127 4558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:39.867262 4558 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:39.869138 4558 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:39.885422 4558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2: Generated new TSK 0
I20260927 22:51:39.886483 4558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:39.933840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33423
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:40.243134 4574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:40.243372 4574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:40.243541 4574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:40.250519 4574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:40.250610 4574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:40.250802 4574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:40.260278 4574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:40.261449 4574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:40.262666 4574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:40.275022 4582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:40.275058 4579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:40.275111 4580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:40.276209 4574 server_base.cc:1034] running on GCE node
I20260927 22:51:40.276762 4574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:40.277709 4574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:40.278936 4574 hybrid_clock.cc:648] HybridClock initialized: now 1790549500278893 us; error 51 us; skew 500 ppm
I20260927 22:51:40.279232 4574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:40 dist-test-slave-df53 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:40.282222 4574 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:40.283295 4574 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:40.283380 4574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:40.283623 4574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:40.286356 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "1ac5bc38609e48c088f534c65f0a6999"
format_stamp: "Formatted at 2026-09-27 22:51:40 on dist-test-slave-df53"
server_key: "ebcb8be9db39662f32ed159e18fe1f05"
server_key_iv: "a8632122a9191e9dc22102f104e4f4a4"
server_key_version: "encryptionkey@0"
I20260927 22:51:40.287081 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1ac5bc38609e48c088f534c65f0a6999"
format_stamp: "Formatted at 2026-09-27 22:51:40 on dist-test-slave-df53"
server_key: "ebcb8be9db39662f32ed159e18fe1f05"
server_key_iv: "a8632122a9191e9dc22102f104e4f4a4"
server_key_version: "encryptionkey@0"
I20260927 22:51:40.291692 4574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:40.294838 4588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:40.296039 4574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:40.296164 4574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "1ac5bc38609e48c088f534c65f0a6999"
format_stamp: "Formatted at 2026-09-27 22:51:40 on dist-test-slave-df53"
server_key: "ebcb8be9db39662f32ed159e18fe1f05"
server_key_iv: "a8632122a9191e9dc22102f104e4f4a4"
server_key_version: "encryptionkey@0"
I20260927 22:51:40.296320 4574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:40.311748 4574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:40.315640 4574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:40.315908 4574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:40.316743 4574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:40.317994 4574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:40.318109 4574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:40.318243 4574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:40.318336 4574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:40.349375 4574 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34705
I20260927 22:51:40.349423 4700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34705 every 8 connection(s)
I20260927 22:51:40.350257 4574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:40.350574 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 4574
I20260927 22:51:40.350805 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:40.351507 388 external_mini_cluster.cc:1428] Setting key c1e1a1c3f1134c0518c73fb432d4352f
Sep 27 22:51:40 dist-test-slave-df53 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:40.370091 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:40.352720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34041 (local address 127.0.97.62:33423)
0927 22:51:40.353220 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:40.353226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:40.354280 (+ 1054us) server_negotiation.cc:407] Connection header received
0927 22:51:40.356280 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:40.356285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:40.356365 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:40.356490 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:40.357963 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:40.359095 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:40.360581 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:40.360988 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:40.361118 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:40.364663 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:40.364667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:40.364671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:40.364698 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:40.367221 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:40.367916 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:40.367928 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:40.367930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:40.368036 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:40.368434 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:40.368439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:40.368442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:40.368689 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:40.368841 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:40.369500 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 22:51:40.369728 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":270,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:51:40.371115 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:40.353409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33423 (local address 127.0.97.1:34041)
0927 22:51:40.354145 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:40.354182 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:40.356027 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:40.356716 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:40.356724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:40.357199 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:40.357806 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:40.357828 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:40.359266 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:40.359270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:40.360455 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:40.360467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:40.361198 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:40.361207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:40.361455 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:40.362180 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:40.362200 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:40.364450 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:40.367358 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:40.367366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:40.367381 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:40.367777 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:40.368140 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:40.368146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:40.368149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:40.368320 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:40.368816 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:40.368823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:40.369156 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 22:51:40.369404 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 22:51:40.369905 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":323,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:51:40.372780 4701 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33423
I20260927 22:51:40.373090 4701 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:40.373775 4701 heartbeater.cc:511] Master 127.0.97.62:33423 requested a full tablet report, sending...
I20260927 22:51:40.375720 4513 ts_manager.cc:194] Registered new tserver with Master: 1ac5bc38609e48c088f534c65f0a6999 (127.0.97.1:34705)
I20260927 22:51:40.377532 4513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:40.397272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33423
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:40.708197 4707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:40.708449 4707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:40.708598 4707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:40.715612 4707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:40.715703 4707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:40.715912 4707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:40.725354 4707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:40.726585 4707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:40.727751 4707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:40.739852 4712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:40.739820 4713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:40.739779 4715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:40.740944 4707 server_base.cc:1034] running on GCE node
I20260927 22:51:40.741520 4707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:40.742359 4707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:40.743561 4707 hybrid_clock.cc:648] HybridClock initialized: now 1790549500743503 us; error 50 us; skew 500 ppm
I20260927 22:51:40.743846 4707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:40 dist-test-slave-df53 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:40.746891 4707 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:40.747963 4707 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:40.748045 4707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:40.748286 4707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:40.750960 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "3f95e68d432244d9a236db31cc2f3218"
format_stamp: "Formatted at 2026-09-27 22:51:40 on dist-test-slave-df53"
server_key: "e9c35d6c91cf5a6c82ac8e376c872a3a"
server_key_iv: "d0032ad4c0bbf6198efd90ccaa48b126"
server_key_version: "encryptionkey@0"
I20260927 22:51:40.751679 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3f95e68d432244d9a236db31cc2f3218"
format_stamp: "Formatted at 2026-09-27 22:51:40 on dist-test-slave-df53"
server_key: "e9c35d6c91cf5a6c82ac8e376c872a3a"
server_key_iv: "d0032ad4c0bbf6198efd90ccaa48b126"
server_key_version: "encryptionkey@0"
I20260927 22:51:40.756738 4707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:51:40.759927 4721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:40.761150 4707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:51:40.761317 4707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "3f95e68d432244d9a236db31cc2f3218"
format_stamp: "Formatted at 2026-09-27 22:51:40 on dist-test-slave-df53"
server_key: "e9c35d6c91cf5a6c82ac8e376c872a3a"
server_key_iv: "d0032ad4c0bbf6198efd90ccaa48b126"
server_key_version: "encryptionkey@0"
I20260927 22:51:40.761547 4707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:40.775694 4707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:40.779753 4707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:40.780032 4707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:40.780928 4707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:40.782196 4707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:40.782315 4707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:40.782452 4707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:40.782539 4707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:40.816953 4707 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39875
I20260927 22:51:40.817004 4833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39875 every 8 connection(s)
I20260927 22:51:40.817848 4707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:40.823318 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 4707
I20260927 22:51:40.823525 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:40.823966 388 external_mini_cluster.cc:1428] Setting key c3e97746bbe57046a886a41d46ad0010
Sep 27 22:51:40 dist-test-slave-df53 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:40.837242 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:40.820156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40619 (local address 127.0.97.62:33423)
0927 22:51:40.820323 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:40.820328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:40.821449 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 22:51:40.823223 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:40.823228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:40.823309 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:40.823432 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:40.825232 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:40.826246 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:40.827692 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:40.828416 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:40.828560 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:40.831932 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:40.831938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:40.831941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:40.831993 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:40.834497 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:40.835137 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:40.835144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:40.835147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:40.835287 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:40.835681 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:40.835687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:40.835690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:40.835935 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:40.836082 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:40.836710 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 22:51:40.836927 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:51:40.838155 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:40.820656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33423 (local address 127.0.97.2:40619)
0927 22:51:40.821281 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:40.821316 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:40.822955 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:40.823709 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:40.823717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:40.824299 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:40.825076 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:40.825099 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:40.826379 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:40.826382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:40.827552 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:40.827563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:40.828609 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:40.828613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:40.828779 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:40.829430 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:40.829455 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:40.831725 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:40.834624 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:40.834632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:40.834646 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:40.835021 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:40.835392 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:40.835405 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:40.835409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:40.835574 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:40.836056 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:40.836064 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:40.836358 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 22:51:40.836613 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 22:51:40.837078 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":256,"threads_started":1}
I20260927 22:51:40.839676 4834 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33423
I20260927 22:51:40.839946 4834 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:40.840528 4834 heartbeater.cc:511] Master 127.0.97.62:33423 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:51:40.841928 4513 ts_manager.cc:194] Registered new tserver with Master: 3f95e68d432244d9a236db31cc2f3218 (127.0.97.2:39875)
I20260927 22:51:40.842779 4513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:40.865036 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33423
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:41.177079 4839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:41.177354 4839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:41.177541 4839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:41.185678 4839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:41.185774 4839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:41.185999 4839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:41.195502 4839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:41.196728 4839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:41.198004 4839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:41.211692 4844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:41.211664 4845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:41.211963 4847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:41.215005 4839 server_base.cc:1034] running on GCE node
I20260927 22:51:41.215554 4839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:41.216442 4839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:41.217684 4839 hybrid_clock.cc:648] HybridClock initialized: now 1790549501217655 us; error 58 us; skew 500 ppm
I20260927 22:51:41.217983 4839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:41 dist-test-slave-df53 krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:41.221002 4839 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:41.222164 4839 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:41.222292 4839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:41.222611 4839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:41.225319 4839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "c2181e38ef7148058f1f1e4f0dbf640e"
format_stamp: "Formatted at 2026-09-27 22:51:41 on dist-test-slave-df53"
server_key: "e6f0acf9149dc12dfa886c65dcd37abc"
server_key_iv: "c93d9e2dbbac47aeea867d6dfaf1d5b9"
server_key_version: "encryptionkey@0"
I20260927 22:51:41.226142 4839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c2181e38ef7148058f1f1e4f0dbf640e"
format_stamp: "Formatted at 2026-09-27 22:51:41 on dist-test-slave-df53"
server_key: "e6f0acf9149dc12dfa886c65dcd37abc"
server_key_iv: "c93d9e2dbbac47aeea867d6dfaf1d5b9"
server_key_version: "encryptionkey@0"
I20260927 22:51:41.230757 4839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:51:41.234025 4853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:41.235157 4839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 22:51:41.235329 4839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "c2181e38ef7148058f1f1e4f0dbf640e"
format_stamp: "Formatted at 2026-09-27 22:51:41 on dist-test-slave-df53"
server_key: "e6f0acf9149dc12dfa886c65dcd37abc"
server_key_iv: "c93d9e2dbbac47aeea867d6dfaf1d5b9"
server_key_version: "encryptionkey@0"
I20260927 22:51:41.235549 4839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:41.261554 4839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:41.265846 4839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:41.266103 4839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:41.266949 4839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:41.268092 4839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:41.268172 4839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:41.268245 4839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:41.268298 4839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:41.300988 4839 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46263
I20260927 22:51:41.301069 4965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46263 every 8 connection(s)
I20260927 22:51:41.301872 4839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:41.311585 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 4839
I20260927 22:51:41.311806 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:41.312343 388 external_mini_cluster.cc:1428] Setting key ccda86d33eb7eb07d0a2464ff6f95096
Sep 27 22:51:41 dist-test-slave-df53 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:41.320827 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.304296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43425 (local address 127.0.97.62:33423)
0927 22:51:41.304490 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.304495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.305612 (+ 1117us) server_negotiation.cc:407] Connection header received
0927 22:51:41.307321 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.307327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.307407 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.307533 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:41.308923 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.309778 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.311278 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.311666 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.311810 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.315887 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:41.315891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:41.315894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:41.315923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:41.318344 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.318989 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.318992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.318995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.319095 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.319534 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.319537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.319539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.319780 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:41.319886 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.320471 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.320608 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:51:41.321789 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.304771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33423 (local address 127.0.97.3:43425)
0927 22:51:41.305414 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:41.305474 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:41.307060 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:41.307757 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:41.307765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:41.308238 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:41.308762 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.308779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.309935 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.309943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.311129 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.311141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.311919 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.311928 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.312199 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.312910 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:41.312931 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:41.315713 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:41.318475 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:41.318481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:41.318492 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:41.318890 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:41.319243 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:41.319248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:41.319251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:41.319443 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:41.319881 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:41.319886 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:41.320145 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 22:51:41.320408 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 22:51:41.320725 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":262,"threads_started":1}
I20260927 22:51:41.322981 4966 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33423
I20260927 22:51:41.323271 4966 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:41.323918 4966 heartbeater.cc:511] Master 127.0.97.62:33423 requested a full tablet report, sending...
I20260927 22:51:41.325244 4513 ts_manager.cc:194] Registered new tserver with Master: c2181e38ef7148058f1f1e4f0dbf640e (127.0.97.3:46263)
I20260927 22:51:41.326241 4513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43425
I20260927 22:51:41.327592 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:51:41.343467 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.331956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.0.97.62:33423)
0927 22:51:41.332109 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.332113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.332627 (+ 514us) server_negotiation.cc:407] Connection header received
0927 22:51:41.332746 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.332750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.332799 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.332874 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:41.334256 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.334916 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.336095 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.336510 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.336596 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.338912 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:41.338916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:41.338919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:41.338953 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:41.341376 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.342014 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.342017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.342020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.342130 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.342430 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.342434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.342437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.342684 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:41.342844 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.343125 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.343261 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:51:41.347093 4513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:41.356134 4513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:41.381114 4701 heartbeater.cc:503] Master 127.0.97.62:33423 was elected leader, sending a full tablet report...
I20260927 22:51:41.390733 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.366997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46263 (local address 127.0.0.1:36542)
0927 22:51:41.380418 (+ 13421us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:41.380466 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:41.380687 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:41.382529 (+ 1842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:41.382540 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:41.382582 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:41.383086 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.383102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.385197 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.385203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.387680 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.387695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.389795 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.389803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.390008 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.390056 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:51:41.390169 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 22:51:41.390350 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13113,"thread_start_us":18024,"threads_started":1}
I20260927 22:51:41.390921 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.367268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.0.97.3:46263)
0927 22:51:41.378256 (+ 10988us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.378269 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.380951 (+ 2682us) server_negotiation.cc:407] Connection header received
0927 22:51:41.381063 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.381069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.381318 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.381503 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:41.383320 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.385030 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.387844 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.389253 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.389397 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.389649 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.390395 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.390570 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10772,"thread_start_us":10605,"threads_started":1}
I20260927 22:51:41.393175 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.369468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.0.97.1:34705)
0927 22:51:41.384162 (+ 14694us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.384171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.385453 (+ 1282us) server_negotiation.cc:407] Connection header received
0927 22:51:41.385659 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.385668 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.385899 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.386088 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:41.387111 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.388314 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.390685 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.392108 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.392281 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.392492 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.392706 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.392851 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14533,"thread_start_us":14682,"threads_started":1}
I20260927 22:51:41.393210 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.369116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34705 (local address 127.0.0.1:53770)
0927 22:51:41.385321 (+ 16205us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:41.385338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:41.385485 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:41.386356 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:41.386360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:41.386384 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:41.386904 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.386919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.388450 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.388455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.390510 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.390524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.392308 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.392314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.392481 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.392516 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:51:41.392666 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 22:51:41.392781 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16056,"mutex_wait_us":71,"thread_start_us":15357,"threads_started":1}
I20260927 22:51:41.393764 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.367419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.0.97.2:39875)
0927 22:51:41.385958 (+ 18539us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.385967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.385995 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:51:41.386088 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.386095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.386307 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.387207 (+ 900us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:41.387414 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.388588 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.391112 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.392192 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.392319 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.392502 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.393302 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.393454 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18374,"thread_start_us":19301,"threads_started":1}
I20260927 22:51:41.394187 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.367307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39875 (local address 127.0.0.1:54144)
0927 22:51:41.384721 (+ 17414us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:41.384739 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:41.384838 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:41.386542 (+ 1704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:41.386547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:41.386577 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:41.387023 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.387034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.388735 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.388742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.390943 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.390955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.392647 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.392653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.392807 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.392851 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:51:41.392957 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:51:41.393079 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17299,"mutex_wait_us":23,"thread_start_us":14469,"threads_started":1}
I20260927 22:51:41.395076 4901 tablet_service.cc:1467] Processing CreateTablet for tablet a453b013c9104ebeb255cb063a0453d9 (DEFAULT_TABLE table=test-table [id=463a2a722e5f4170b3186456d6beca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:41.396147 4901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a453b013c9104ebeb255cb063a0453d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:41.396243 4636 tablet_service.cc:1467] Processing CreateTablet for tablet a453b013c9104ebeb255cb063a0453d9 (DEFAULT_TABLE table=test-table [id=463a2a722e5f4170b3186456d6beca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:41.397153 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a453b013c9104ebeb255cb063a0453d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:41.400777 4769 tablet_service.cc:1467] Processing CreateTablet for tablet a453b013c9104ebeb255cb063a0453d9 (DEFAULT_TABLE table=test-table [id=463a2a722e5f4170b3186456d6beca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:41.401979 4769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a453b013c9104ebeb255cb063a0453d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:41.408036 4984 tablet_bootstrap.cc:492] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: Bootstrap starting.
I20260927 22:51:41.411067 4985 tablet_bootstrap.cc:492] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: Bootstrap starting.
I20260927 22:51:41.411450 4984 tablet_bootstrap.cc:654] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:41.412837 4984 log.cc:824] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:41.414752 4985 tablet_bootstrap.cc:654] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:41.414965 4987 tablet_bootstrap.cc:492] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: Bootstrap starting.
I20260927 22:51:41.416043 4985 log.cc:824] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:41.416204 4984 tablet_bootstrap.cc:492] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: No bootstrap required, opened a new log
I20260927 22:51:41.416462 4984 ts_tablet_manager.cc:1397] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:51:41.417872 4987 tablet_bootstrap.cc:654] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:41.418722 4987 log.cc:824] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:41.418978 4985 tablet_bootstrap.cc:492] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: No bootstrap required, opened a new log
I20260927 22:51:41.419209 4985 ts_tablet_manager.cc:1397] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260927 22:51:41.426234 4987 tablet_bootstrap.cc:492] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: No bootstrap required, opened a new log
I20260927 22:51:41.426463 4987 ts_tablet_manager.cc:1397] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260927 22:51:41.426661 4984 raft_consensus.cc:348] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 } } peers { permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 } } peers { permanent_uuid: "1ac5bc38609e48c088f534c65f0a6999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } }
I20260927 22:51:41.427057 4984 raft_consensus.cc:374] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:41.427147 4984 raft_consensus.cc:729] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac5bc38609e48c088f534c65f0a6999, State: Initialized, Role: FOLLOWER
I20260927 22:51:41.427735 4984 consensus_queue.cc:260] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 } } peers { permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 } } peers { permanent_uuid: "1ac5bc38609e48c088f534c65f0a6999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } }
I20260927 22:51:41.429406 4984 ts_tablet_manager.cc:1428] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 22:51:41.429464 4985 raft_consensus.cc:348] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 } } peers { permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 } } peers { permanent_uuid: "1ac5bc38609e48c088f534c65f0a6999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } }
I20260927 22:51:41.429945 4985 raft_consensus.cc:374] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:41.430084 4985 raft_consensus.cc:729] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2181e38ef7148058f1f1e4f0dbf640e, State: Initialized, Role: FOLLOWER
I20260927 22:51:41.430789 4985 consensus_queue.cc:260] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 } } peers { permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 } } peers { permanent_uuid: "1ac5bc38609e48c088f534c65f0a6999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } }
I20260927 22:51:41.431994 4966 heartbeater.cc:503] Master 127.0.97.62:33423 was elected leader, sending a full tablet report...
I20260927 22:51:41.432600 4985 ts_tablet_manager.cc:1428] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260927 22:51:41.435521 4987 raft_consensus.cc:348] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 } } peers { permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 } } peers { permanent_uuid: "1ac5bc38609e48c088f534c65f0a6999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } }
I20260927 22:51:41.435941 4987 raft_consensus.cc:374] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:41.436051 4987 raft_consensus.cc:729] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f95e68d432244d9a236db31cc2f3218, State: Initialized, Role: FOLLOWER
I20260927 22:51:41.436559 4987 consensus_queue.cc:260] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 } } peers { permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 } } peers { permanent_uuid: "1ac5bc38609e48c088f534c65f0a6999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } }
I20260927 22:51:41.437383 4834 heartbeater.cc:503] Master 127.0.97.62:33423 was elected leader, sending a full tablet report...
I20260927 22:51:41.437883 4987 ts_tablet_manager.cc:1428] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 22:51:41.490052 4990 raft_consensus.cc:482] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:41.490346 4990 raft_consensus.cc:504] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 } } peers { permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 } } peers { permanent_uuid: "1ac5bc38609e48c088f534c65f0a6999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } }
I20260927 22:51:41.492058 4990 leader_election.cc:288] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2181e38ef7148058f1f1e4f0dbf640e (127.0.97.3:46263), 3f95e68d432244d9a236db31cc2f3218 (127.0.97.2:39875)
I20260927 22:51:41.499159 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.492438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57839 (local address 127.0.97.3:46263)
0927 22:51:41.492681 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.492690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.493409 (+ 719us) server_negotiation.cc:407] Connection header received
0927 22:51:41.493650 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.493655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.493713 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.493812 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:41.494443 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.495144 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.497408 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.498221 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.498368 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.498473 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.498783 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.498899 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:51:41.499195 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.492548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49447 (local address 127.0.97.2:39875)
0927 22:51:41.492768 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.492777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.493384 (+ 607us) server_negotiation.cc:407] Connection header received
0927 22:51:41.493605 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.493610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.493706 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.493801 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:41.494367 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.495057 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.497419 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.498229 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.498394 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.498492 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.498808 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.498930 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:51:41.499310 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.492358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39875 (local address 127.0.97.1:49447)
0927 22:51:41.493248 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:41.493263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:41.493420 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:41.493814 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:41.493819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:41.493850 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:41.494217 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.494228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.495189 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.495197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.497273 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.497284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.498468 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.498474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.498652 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.498674 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:51:41.498771 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:51:41.498903 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":79,"thread_start_us":652,"threads_started":1}
I20260927 22:51:41.499518 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.492374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46263 (local address 127.0.97.1:57839)
0927 22:51:41.493297 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:41.493311 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:41.493458 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:41.493823 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:41.493827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:41.493910 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:41.494272 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.494280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.495278 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.495285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.497262 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:41.497273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.498440 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.498450 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:41.498630 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.498653 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:51:41.498751 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:51:41.498872 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":3,"spinlock_wait_cycles":24832,"thread_start_us":540,"threads_started":1}
I20260927 22:51:41.500259 4789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a453b013c9104ebeb255cb063a0453d9" candidate_uuid: "1ac5bc38609e48c088f534c65f0a6999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f95e68d432244d9a236db31cc2f3218" is_pre_election: true
I20260927 22:51:41.500473 4921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a453b013c9104ebeb255cb063a0453d9" candidate_uuid: "1ac5bc38609e48c088f534c65f0a6999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" is_pre_election: true
I20260927 22:51:41.500751 4789 raft_consensus.cc:2454] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ac5bc38609e48c088f534c65f0a6999 in term 0.
I20260927 22:51:41.500991 4921 raft_consensus.cc:2454] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ac5bc38609e48c088f534c65f0a6999 in term 0.
I20260927 22:51:41.501402 4589 leader_election.cc:302] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac5bc38609e48c088f534c65f0a6999, 3f95e68d432244d9a236db31cc2f3218; no voters:
I20260927 22:51:41.501806 4990 raft_consensus.cc:2781] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:41.501951 4990 raft_consensus.cc:482] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:41.502060 4990 raft_consensus.cc:3037] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:41.504331 4990 raft_consensus.cc:504] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 } } peers { permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 } } peers { permanent_uuid: "1ac5bc38609e48c088f534c65f0a6999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } }
I20260927 22:51:41.504851 4990 leader_election.cc:288] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [CANDIDATE]: Term 1 election: Requested vote from peers c2181e38ef7148058f1f1e4f0dbf640e (127.0.97.3:46263), 3f95e68d432244d9a236db31cc2f3218 (127.0.97.2:39875)
I20260927 22:51:41.505365 4789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a453b013c9104ebeb255cb063a0453d9" candidate_uuid: "1ac5bc38609e48c088f534c65f0a6999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f95e68d432244d9a236db31cc2f3218"
I20260927 22:51:41.505405 4921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a453b013c9104ebeb255cb063a0453d9" candidate_uuid: "1ac5bc38609e48c088f534c65f0a6999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2181e38ef7148058f1f1e4f0dbf640e"
I20260927 22:51:41.505570 4789 raft_consensus.cc:3037] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:41.505574 4921 raft_consensus.cc:3037] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:41.508685 4921 raft_consensus.cc:2454] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ac5bc38609e48c088f534c65f0a6999 in term 1.
I20260927 22:51:41.508685 4789 raft_consensus.cc:2454] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ac5bc38609e48c088f534c65f0a6999 in term 1.
I20260927 22:51:41.509106 4589 leader_election.cc:302] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac5bc38609e48c088f534c65f0a6999, 3f95e68d432244d9a236db31cc2f3218; no voters:
I20260927 22:51:41.509395 4990 raft_consensus.cc:2781] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:41.509891 4990 raft_consensus.cc:686] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 1 LEADER]: Becoming Leader. State: Replica: 1ac5bc38609e48c088f534c65f0a6999, State: Running, Role: LEADER
I20260927 22:51:41.510344 4990 consensus_queue.cc:237] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 } } peers { permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 } } peers { permanent_uuid: "1ac5bc38609e48c088f534c65f0a6999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } }
I20260927 22:51:41.514710 4513 catalog_manager.cc:5495] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ac5bc38609e48c088f534c65f0a6999 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1ac5bc38609e48c088f534c65f0a6999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ac5bc38609e48c088f534c65f0a6999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34705 } health_report { overall_health: HEALTHY } } }
I20260927 22:51:41.543619 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.538516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.0.97.1:34705)
0927 22:51:41.538700 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.538705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.538773 (+ 68us) server_negotiation.cc:407] Connection header received
0927 22:51:41.538970 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.538974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.539042 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.539212 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:41.539575 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.540321 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.541647 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.542038 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.542165 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.542605 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:41.543063 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:41.543220 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.543299 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.543370 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:41.551502 4789 raft_consensus.cc:1261] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218 [term 1 FOLLOWER]: Refusing update from remote peer 1ac5bc38609e48c088f534c65f0a6999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:41.551502 4921 raft_consensus.cc:1261] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e [term 1 FOLLOWER]: Refusing update from remote peer 1ac5bc38609e48c088f534c65f0a6999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:41.552419 4995 consensus_queue.cc:1035] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2181e38ef7148058f1f1e4f0dbf640e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:41.552783 4990 consensus_queue.cc:1035] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f95e68d432244d9a236db31cc2f3218" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:51:41.553285 4967 tablet.cc:2369] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:41.563566 4998 mvcc.cc:205] Tried to move back new op lower bound from 7334090758344384512 to 7334090758189703168. Current Snapshot: MvccSnapshot[applied={T|T < 7334090758344384512}]
I20260927 22:51:41.564432 4999 mvcc.cc:205] Tried to move back new op lower bound from 7334090758344384512 to 7334090758189703168. Current Snapshot: MvccSnapshot[applied={T|T < 7334090758344384512}]
I20260927 22:51:41.567006 5000 mvcc.cc:205] Tried to move back new op lower bound from 7334090758344384512 to 7334090758189703168. Current Snapshot: MvccSnapshot[applied={T|T < 7334090758344384512}]
I20260927 22:51:41.577814 4513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:41.578102 4513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:41.581406 4513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0ceffa4caf64c46ba3e8f5f3ceb51d2: Sending DeleteTablet for 3 replicas of tablet a453b013c9104ebeb255cb063a0453d9
I20260927 22:51:41.582583 4769 tablet_service.cc:1514] Processing DeleteTablet for tablet a453b013c9104ebeb255cb063a0453d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54144
I20260927 22:51:41.582640 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet a453b013c9104ebeb255cb063a0453d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53770
I20260927 22:51:41.582665 4901 tablet_service.cc:1514] Processing DeleteTablet for tablet a453b013c9104ebeb255cb063a0453d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36542
I20260927 22:51:41.588680 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 4574
I20260927 22:51:41.607736 5007 tablet_replica.cc:330] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: stopping tablet replica
I20260927 22:51:41.608131 5007 raft_consensus.cc:2229] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:41.608675 5007 raft_consensus.cc:2258] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:41.609419 5009 tablet_replica.cc:330] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: stopping tablet replica
I20260927 22:51:41.609822 5009 raft_consensus.cc:2229] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:41.610291 5009 raft_consensus.cc:2258] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:41.610481 5008 tablet_replica.cc:330] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: stopping tablet replica
I20260927 22:51:41.611037 5008 raft_consensus.cc:2229] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:41.611804 5008 raft_consensus.cc:2258] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:41.613310 5007 ts_tablet_manager.cc:1905] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:41.613708 5008 ts_tablet_manager.cc:1905] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:51:41 dist-test-slave-df53 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:41.621063 5007 ts_tablet_manager.cc:1918] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:41.621204 5008 ts_tablet_manager.cc:1918] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:41.621343 5008 log.cc:1196] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/a453b013c9104ebeb255cb063a0453d9
I20260927 22:51:41.621374 5007 log.cc:1196] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/a453b013c9104ebeb255cb063a0453d9
I20260927 22:51:41.621901 5008 ts_tablet_manager.cc:1939] T a453b013c9104ebeb255cb063a0453d9 P 3f95e68d432244d9a236db31cc2f3218: Deleting consensus metadata
I20260927 22:51:41.622108 5007 ts_tablet_manager.cc:1939] T a453b013c9104ebeb255cb063a0453d9 P 1ac5bc38609e48c088f534c65f0a6999: Deleting consensus metadata
I20260927 22:51:41.622853 5009 ts_tablet_manager.cc:1905] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:41.623641 4497 catalog_manager.cc:4841] TS 3f95e68d432244d9a236db31cc2f3218 (127.0.97.2:39875): tablet a453b013c9104ebeb255cb063a0453d9 (table test-table [id=463a2a722e5f4170b3186456d6beca92]) successfully deleted
I20260927 22:51:41.623997 4500 catalog_manager.cc:4841] TS 1ac5bc38609e48c088f534c65f0a6999 (127.0.97.1:34705): tablet a453b013c9104ebeb255cb063a0453d9 (table test-table [id=463a2a722e5f4170b3186456d6beca92]) successfully deleted
I20260927 22:51:41.625686 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.589939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.0.97.1:34705)
0927 22:51:41.590108 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.590114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.606152 (+ 16038us) server_negotiation.cc:407] Connection header received
0927 22:51:41.606393 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.606399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.606491 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.606601 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:41.609293 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.610433 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.612498 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.613216 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.613324 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.618490 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:41.618503 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:41.618520 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:41.618565 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:41.622607 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.623405 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.623416 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.623419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.623571 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.624004 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.624009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.624012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.624408 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:41.624707 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.625222 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.625374 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:51:41.626384 4696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:41.630956 5009 ts_tablet_manager.cc:1918] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:41.631301 5009 log.cc:1196] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/a453b013c9104ebeb255cb063a0453d9
I20260927 22:51:41.632071 5009 ts_tablet_manager.cc:1939] T a453b013c9104ebeb255cb063a0453d9 P c2181e38ef7148058f1f1e4f0dbf640e: Deleting consensus metadata
I20260927 22:51:41.634109 4499 catalog_manager.cc:4841] TS c2181e38ef7148058f1f1e4f0dbf640e (127.0.97.3:46263): tablet a453b013c9104ebeb255cb063a0453d9 (table test-table [id=463a2a722e5f4170b3186456d6beca92]) successfully deleted
I20260927 22:51:41.724643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 4574
I20260927 22:51:41.741771 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 4707
Sep 27 22:51:41 dist-test-slave-df53 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:41.755514 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.742532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.0.97.2:39875)
0927 22:51:41.742713 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.742718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.742737 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:51:41.742869 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.742872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.742956 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.743044 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:41.744109 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.744858 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.746152 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.746522 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.746617 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.750324 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:41.750337 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:41.750352 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:41.750389 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:41.752915 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.753482 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.753489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.753491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.753676 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.754038 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.754042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.754044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.754447 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:41.754663 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.754978 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.755133 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:41.756242 4829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:41.842746 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 4707
I20260927 22:51:41.858587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 4839
Sep 27 22:51:41 dist-test-slave-df53 krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:51:41.872433 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:41.859338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.0.97.3:46263)
0927 22:51:41.859530 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:41.859534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:41.859552 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:51:41.859709 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:41.859713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:41.859772 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:41.859854 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:41.860994 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.861757 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.862920 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:41.863297 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:41.863402 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:41.867477 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:41.867490 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:41.867505 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:41.867544 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:41.870064 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.870633 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.870640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.870642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.870771 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:41.871076 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:41.871078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:41.871080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:41.871408 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:41.871601 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:41.871907 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 22:51:41.872109 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:51:41.873076 4961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:41.959801 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 4839
I20260927 22:51:41.975514 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 4482
I20260927 22:51:41.976588 4543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:42.081158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 4482
2026-09-27T22:51:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3943 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T22:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:42Z Disabled control of system clock
I20260927 22:51:42.144932 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40659
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:51:42.467940 5018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:42.468202 5018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:42.468282 5018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:42.475885 5018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:42.475982 5018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:42.476018 5018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:42.476064 5018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:42.476105 5018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:42.485038 5018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40659
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:42.486202 5018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:42.487810 5018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:42.499092 5023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:42.499178 5024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:42.499141 5026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:42.500047 5018 server_base.cc:1034] running on GCE node
I20260927 22:51:42.500712 5018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:42.502022 5018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:42.503221 5018 hybrid_clock.cc:648] HybridClock initialized: now 1790549502503163 us; error 50 us; skew 500 ppm
I20260927 22:51:42.503615 5018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:42.505824 5018 webserver.cc:460] Webserver started at http://127.0.97.62:43869/ using document root <none> and password file <none>
I20260927 22:51:42.506474 5018 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:42.506584 5018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:42.506924 5018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:42.509667 5018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "3b407de795a14df08603b26540e7a896"
format_stamp: "Formatted at 2026-09-27 22:51:42 on dist-test-slave-df53"
server_key: "cd84ef1c60367e5577152bb5e741453b"
server_key_iv: "b24aba26baf9e9b86ac82b855db88f0e"
server_key_version: "encryptionkey@0"
I20260927 22:51:42.510511 5018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "3b407de795a14df08603b26540e7a896"
format_stamp: "Formatted at 2026-09-27 22:51:42 on dist-test-slave-df53"
server_key: "cd84ef1c60367e5577152bb5e741453b"
server_key_iv: "b24aba26baf9e9b86ac82b855db88f0e"
server_key_version: "encryptionkey@0"
I20260927 22:51:42.515748 5018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:51:42.518975 5032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:42.520305 5018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:42.520507 5018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "3b407de795a14df08603b26540e7a896"
format_stamp: "Formatted at 2026-09-27 22:51:42 on dist-test-slave-df53"
server_key: "cd84ef1c60367e5577152bb5e741453b"
server_key_iv: "b24aba26baf9e9b86ac82b855db88f0e"
server_key_version: "encryptionkey@0"
I20260927 22:51:42.520711 5018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:42.526114 5018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:42.526804 5018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:42.527102 5018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:42.544299 5018 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40659
I20260927 22:51:42.544353 5083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40659 every 8 connection(s)
I20260927 22:51:42.545524 5018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:42.549132 5084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:42.550871 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 5018
I20260927 22:51:42.551055 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:42.551645 388 external_mini_cluster.cc:1428] Setting key e7aec5364a1c547f5d3f019fcd6b6f11
I20260927 22:51:42.559681 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:42.553761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.0.97.62:40659)
0927 22:51:42.554821 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:42.554831 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:42.554872 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:51:42.556376 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:42.556397 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:42.556756 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:42.557063 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:42.558175 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:42.558181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:42.558192 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:42.558205 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:42.558262 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:42.558280 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:42.558493 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:42.558593 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 22:51:42.558873 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":374,"threads_started":1}
I20260927 22:51:42.560747 5084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896: Bootstrap starting.
I20260927 22:51:42.563841 5084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:42.564745 5084 log.cc:824] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:42.567443 5084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896: No bootstrap required, opened a new log
I20260927 22:51:42.574285 5084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b407de795a14df08603b26540e7a896" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40659 } }
I20260927 22:51:42.574615 5084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:42.574728 5084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b407de795a14df08603b26540e7a896, State: Initialized, Role: FOLLOWER
I20260927 22:51:42.575254 5084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b407de795a14df08603b26540e7a896" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40659 } }
I20260927 22:51:42.575536 5084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:42.575680 5084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:42.575816 5084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:42.578037 5084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b407de795a14df08603b26540e7a896" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40659 } }
I20260927 22:51:42.578519 5084 leader_election.cc:302] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b407de795a14df08603b26540e7a896; no voters:
I20260927 22:51:42.579097 5084 leader_election.cc:288] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:42.579246 5089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:42.579989 5089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [term 1 LEADER]: Becoming Leader. State: Replica: 3b407de795a14df08603b26540e7a896, State: Running, Role: LEADER
I20260927 22:51:42.580523 5089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b407de795a14df08603b26540e7a896" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40659 } }
I20260927 22:51:42.581157 5084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:42.583230 5091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b407de795a14df08603b26540e7a896. Latest consensus state: current_term: 1 leader_uuid: "3b407de795a14df08603b26540e7a896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b407de795a14df08603b26540e7a896" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40659 } } }
I20260927 22:51:42.583732 5091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:42.583588 5090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b407de795a14df08603b26540e7a896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b407de795a14df08603b26540e7a896" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40659 } } }
I20260927 22:51:42.583890 5090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:42.586550 5097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:42.591280 5097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:42.599941 5097 catalog_manager.cc:1347] Generated new cluster ID: 3ce64c03f77f41d992d36ac4b4cc8362
I20260927 22:51:42.600083 5097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:42.616429 5097 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:42.616927 5097 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:42.622956 5097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896: Generated new TSK 0
I20260927 22:51:42.623685 5097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:51:42.629674 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40659
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:51:42.942196 5108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:42.942456 5108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:42.942625 5108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:42.949687 5108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:42.949774 5108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:42.949978 5108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:42.958717 5108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:42.959918 5108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:42.961637 5108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:42.973999 5114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:42.974117 5116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:42.974154 5113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:42.975051 5108 server_base.cc:1034] running on GCE node
I20260927 22:51:42.975643 5108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:42.976603 5108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:42.977875 5108 hybrid_clock.cc:648] HybridClock initialized: now 1790549502977827 us; error 34 us; skew 500 ppm
I20260927 22:51:42.978145 5108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:42.980150 5108 webserver.cc:460] Webserver started at http://127.0.97.1:34729/ using document root <none> and password file <none>
I20260927 22:51:42.980747 5108 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:42.980829 5108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:42.981166 5108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:42.983877 5108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "37eaac87a19f4842ae411849f0c44cd8"
format_stamp: "Formatted at 2026-09-27 22:51:42 on dist-test-slave-df53"
server_key: "9d18edaf0aa2a1b8c19be5174a52a5c3"
server_key_iv: "004a8beb5b9955fe7fad08d58c9361eb"
server_key_version: "encryptionkey@0"
I20260927 22:51:42.984622 5108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "37eaac87a19f4842ae411849f0c44cd8"
format_stamp: "Formatted at 2026-09-27 22:51:42 on dist-test-slave-df53"
server_key: "9d18edaf0aa2a1b8c19be5174a52a5c3"
server_key_iv: "004a8beb5b9955fe7fad08d58c9361eb"
server_key_version: "encryptionkey@0"
I20260927 22:51:42.989537 5108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:51:42.992754 5122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:42.994012 5108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:42.994167 5108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "37eaac87a19f4842ae411849f0c44cd8"
format_stamp: "Formatted at 2026-09-27 22:51:42 on dist-test-slave-df53"
server_key: "9d18edaf0aa2a1b8c19be5174a52a5c3"
server_key_iv: "004a8beb5b9955fe7fad08d58c9361eb"
server_key_version: "encryptionkey@0"
I20260927 22:51:42.994387 5108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:42.999833 5108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:43.000554 5108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:43.000813 5108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:43.001730 5108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:43.003005 5108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:43.003087 5108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.003234 5108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:43.003335 5108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.035252 5108 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33849
I20260927 22:51:43.035326 5234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33849 every 8 connection(s)
I20260927 22:51:43.036577 5108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:43.043955 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.038966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33871 (local address 127.0.97.62:40659)
0927 22:51:43.039123 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:43.039128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:43.040468 (+ 1340us) server_negotiation.cc:407] Connection header received
0927 22:51:43.042131 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:43.042135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:43.042186 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:43.042277 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:43.043422 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:43.043426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:43.043429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:43.043442 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:43.043466 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:43.043476 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:43.043561 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:43.043717 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 22:51:43.043761 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:51:43.044803 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.039493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40659 (local address 127.0.97.1:33871)
0927 22:51:43.040319 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:43.040354 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:43.041863 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:43.042466 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:43.042474 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:43.042561 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:43.043161 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:43.043163 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:43.043198 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:43.043284 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:43.043555 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:43.043562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:43.043567 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:51:43.043763 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 22:51:43.043946 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":250,"threads_started":1}
I20260927 22:51:43.046265 5235 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40659
I20260927 22:51:43.046561 5235 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:43.046708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 5108
I20260927 22:51:43.046896 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:43.047200 5235 heartbeater.cc:511] Master 127.0.97.62:40659 requested a full tablet report, sending...
I20260927 22:51:43.047555 388 external_mini_cluster.cc:1428] Setting key b732c78520888b92ebb1cf3d60788fe9
I20260927 22:51:43.049165 5049 ts_manager.cc:194] Registered new tserver with Master: 37eaac87a19f4842ae411849f0c44cd8 (127.0.97.1:33849)
I20260927 22:51:43.050272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40659
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:51:43.359723 5238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:43.359997 5238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:43.360145 5238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:43.367643 5238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:43.367733 5238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:43.367918 5238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:43.377287 5238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:43.378574 5238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:43.380239 5238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:43.396827 5244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:43.397487 5243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:43.398056 5238 server_base.cc:1034] running on GCE node
W20260927 22:51:43.398455 5246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:43.399067 5238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:43.400009 5238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:43.401214 5238 hybrid_clock.cc:648] HybridClock initialized: now 1790549503401169 us; error 34 us; skew 500 ppm
I20260927 22:51:43.401571 5238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:43.403733 5238 webserver.cc:460] Webserver started at http://127.0.97.2:44577/ using document root <none> and password file <none>
I20260927 22:51:43.404398 5238 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:43.404507 5238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:43.404810 5238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:43.407620 5238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "ce18e2f83e15415b96a6668505857f35"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-df53"
server_key: "c1d62ada0cf8f7406c8e0fcccced27b8"
server_key_iv: "86d26ab26602fc631c36e36da3fee1f1"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.408382 5238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ce18e2f83e15415b96a6668505857f35"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-df53"
server_key: "c1d62ada0cf8f7406c8e0fcccced27b8"
server_key_iv: "86d26ab26602fc631c36e36da3fee1f1"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.413173 5238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 22:51:43.416400 5252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.417737 5238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:43.417893 5238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "ce18e2f83e15415b96a6668505857f35"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-df53"
server_key: "c1d62ada0cf8f7406c8e0fcccced27b8"
server_key_iv: "86d26ab26602fc631c36e36da3fee1f1"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.418113 5238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:43.423599 5238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:43.424311 5238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:43.424597 5238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:43.425436 5238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:43.426680 5238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:43.426789 5238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.426928 5238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:43.427017 5238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.458017 5238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34139
I20260927 22:51:43.458068 5364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34139 every 8 connection(s)
I20260927 22:51:43.459277 5238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:43.466768 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.461711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44057 (local address 127.0.97.62:40659)
0927 22:51:43.461909 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:43.461916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:43.463264 (+ 1348us) server_negotiation.cc:407] Connection header received
0927 22:51:43.464881 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:43.464886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:43.464934 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:43.465021 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:43.466193 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:43.466197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:43.466200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:43.466214 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:43.466243 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:43.466254 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:43.466343 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:43.466511 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 22:51:43.466556 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:51:43.467547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 5238
I20260927 22:51:43.467634 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.462267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40659 (local address 127.0.97.2:44057)
0927 22:51:43.463127 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:43.463162 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:43.464652 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:43.465204 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:43.465212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:43.465294 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:43.465936 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:43.465938 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:43.465972 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:43.466048 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:43.466335 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:43.466342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:43.466358 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 22:51:43.466555 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 22:51:43.466728 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":243,"threads_started":1}
I20260927 22:51:43.467686 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:43.468163 388 external_mini_cluster.cc:1428] Setting key ebfc00f026d2dd6a46a425e6e6c70d92
I20260927 22:51:43.468948 5365 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40659
I20260927 22:51:43.469219 5365 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:43.469828 5365 heartbeater.cc:511] Master 127.0.97.62:40659 requested a full tablet report, sending...
I20260927 22:51:43.470642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40659
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:51:43.471102 5049 ts_manager.cc:194] Registered new tserver with Master: ce18e2f83e15415b96a6668505857f35 (127.0.97.2:34139)
W20260927 22:51:43.788899 5368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:43.789145 5368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:43.789228 5368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:43.796277 5368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:43.796362 5368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:43.796476 5368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:43.805256 5368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:43.806445 5368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:43.808089 5368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:43.820663 5373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:43.820583 5376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:43.820631 5374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:43.821655 5368 server_base.cc:1034] running on GCE node
I20260927 22:51:43.822222 5368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:43.823093 5368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:43.824301 5368 hybrid_clock.cc:648] HybridClock initialized: now 1790549503824254 us; error 55 us; skew 500 ppm
I20260927 22:51:43.824642 5368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:43.826761 5368 webserver.cc:460] Webserver started at http://127.0.97.3:39833/ using document root <none> and password file <none>
I20260927 22:51:43.827414 5368 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:43.827526 5368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:43.827818 5368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:43.830451 5368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "dc869d5c57c5432385a1d6aa98f2d22f"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-df53"
server_key: "6b9803e58ef9f093f87993e6653629e4"
server_key_iv: "6b1a9b1fd712f092ade49db5886084d6"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.831212 5368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dc869d5c57c5432385a1d6aa98f2d22f"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-df53"
server_key: "6b9803e58ef9f093f87993e6653629e4"
server_key_iv: "6b1a9b1fd712f092ade49db5886084d6"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.835872 5368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:43.839181 5382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.840390 5368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:51:43.840560 5368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "dc869d5c57c5432385a1d6aa98f2d22f"
format_stamp: "Formatted at 2026-09-27 22:51:43 on dist-test-slave-df53"
server_key: "6b9803e58ef9f093f87993e6653629e4"
server_key_iv: "6b1a9b1fd712f092ade49db5886084d6"
server_key_version: "encryptionkey@0"
I20260927 22:51:43.840792 5368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:43.846278 5368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:43.847004 5368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:43.847257 5368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:43.848119 5368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:43.849468 5368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:43.849584 5368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.849771 5368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:43.849862 5368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:43.881359 5368 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40487
I20260927 22:51:43.881441 5494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40487 every 8 connection(s)
I20260927 22:51:43.882695 5368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:43.887846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 5368
I20260927 22:51:43.888011 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:43.888438 388 external_mini_cluster.cc:1428] Setting key 41b229cfa4d3dab9d253b9cc4f1c03ce
I20260927 22:51:43.890954 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.885093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34935 (local address 127.0.97.62:40659)
0927 22:51:43.885262 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:43.885266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:43.886693 (+ 1427us) server_negotiation.cc:407] Connection header received
0927 22:51:43.888396 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:43.888401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:43.888469 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:43.888578 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:43.890208 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:43.890213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:43.890218 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:43.890239 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:43.890281 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:43.890302 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:43.890398 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:43.890625 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 22:51:43.890679 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:51:43.891865 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.885652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40659 (local address 127.0.97.3:34935)
0927 22:51:43.886520 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:43.886560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:43.888153 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:43.888879 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:43.888891 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:43.889017 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:43.889842 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:43.889845 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:43.889895 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:43.890006 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:43.890404 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:43.890414 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:43.890423 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:51:43.890661 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 22:51:43.890842 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":262,"threads_started":1}
I20260927 22:51:43.893034 5495 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40659
I20260927 22:51:43.893316 5495 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:43.893894 5495 heartbeater.cc:511] Master 127.0.97.62:40659 requested a full tablet report, sending...
I20260927 22:51:43.895093 5049 ts_manager.cc:194] Registered new tserver with Master: dc869d5c57c5432385a1d6aa98f2d22f (127.0.97.3:40487)
I20260927 22:51:43.903466 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:51:43.910002 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.907745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.0.97.62:40659)
0927 22:51:43.907947 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:43.907956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:43.908298 (+ 342us) server_negotiation.cc:407] Connection header received
0927 22:51:43.908460 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:43.908463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:43.908510 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:43.908596 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:43.909376 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:43.909380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:43.909383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:43.909397 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:43.909424 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:43.909434 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:43.909530 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:43.909663 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 22:51:43.909708 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:51:43.913528 5049 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:43.919981 5049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:43.937057 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.931469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33849 (local address 127.0.0.1:34582)
0927 22:51:43.932987 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:43.933002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:43.933080 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:43.934121 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:43.934129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:43.934232 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:43.935130 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:43.935136 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:43.935166 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:43.935244 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:43.936276 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:43.936286 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:43.936293 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:51:43.936468 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 22:51:43.936552 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"thread_start_us":1288,"threads_started":1}
I20260927 22:51:43.937400 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.932777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.0.97.2:34139)
0927 22:51:43.934062 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:43.934074 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:43.934850 (+ 776us) server_negotiation.cc:407] Connection header received
0927 22:51:43.934921 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:43.934930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:43.935147 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:43.935293 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:43.936589 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:43.936601 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:43.936611 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:43.936632 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:43.936671 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:43.936685 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:43.936859 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:43.936927 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 22:51:43.937015 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":968,"threads_started":1}
I20260927 22:51:43.937551 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.930870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40487 (local address 127.0.0.1:37252)
0927 22:51:43.932308 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:43.932363 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:43.932510 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:43.934121 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:43.934129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:43.934233 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:43.935409 (+ 1176us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:43.935412 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:43.935427 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:43.935469 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:43.936842 (+ 1373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:43.936863 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:43.936868 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:51:43.937002 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 22:51:43.937065 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"thread_start_us":338,"threads_started":1}
I20260927 22:51:43.937530 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.932167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34139 (local address 127.0.0.1:59730)
0927 22:51:43.934545 (+ 2378us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:43.934581 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:43.934702 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:43.935660 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:43.935667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:43.935694 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:43.936404 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:43.936407 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:43.936418 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:43.936453 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:43.936783 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:43.936787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:43.936792 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:51:43.936936 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:51:43.936986 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2281,"thread_start_us":2071,"threads_started":1}
I20260927 22:51:43.938377 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.931110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.0.97.3:40487)
0927 22:51:43.933636 (+ 2526us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:43.933643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:43.933669 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:51:43.933724 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:43.933729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:43.933924 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:43.934550 (+ 626us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:43.936615 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:43.936622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:43.936635 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:43.936653 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:43.936701 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:43.936720 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:43.937870 (+ 1150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:43.937938 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 22:51:43.938023 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2355,"thread_start_us":2338,"threads_started":1}
I20260927 22:51:43.938417 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:43.931662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.0.97.1:33849)
0927 22:51:43.933563 (+ 1901us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:43.933571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:43.933631 (+ 60us) server_negotiation.cc:407] Connection header received
0927 22:51:43.933703 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:43.933710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:43.933926 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:43.934798 (+ 872us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:43.936028 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:43.936036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:43.936051 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:43.936070 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:43.936124 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:43.936150 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:43.937984 (+ 1834us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:43.938037 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 22:51:43.938115 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1758,"thread_start_us":1625,"threads_started":1}
I20260927 22:51:43.941519 5170 tablet_service.cc:1467] Processing CreateTablet for tablet 64002a8854cd44f7af093babbcc44709 (DEFAULT_TABLE table=test-table [id=0b482ef7dade444090e007ffdd5094f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:43.941620 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 64002a8854cd44f7af093babbcc44709 (DEFAULT_TABLE table=test-table [id=0b482ef7dade444090e007ffdd5094f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:43.942003 5430 tablet_service.cc:1467] Processing CreateTablet for tablet 64002a8854cd44f7af093babbcc44709 (DEFAULT_TABLE table=test-table [id=0b482ef7dade444090e007ffdd5094f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:43.942775 5170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64002a8854cd44f7af093babbcc44709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:43.943018 5300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64002a8854cd44f7af093babbcc44709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:43.943795 5430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64002a8854cd44f7af093babbcc44709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:43.953352 5513 tablet_bootstrap.cc:492] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8: Bootstrap starting.
I20260927 22:51:43.956773 5513 tablet_bootstrap.cc:654] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:43.956950 5514 tablet_bootstrap.cc:492] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: Bootstrap starting.
I20260927 22:51:43.958182 5513 log.cc:824] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:43.959556 5515 tablet_bootstrap.cc:492] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: Bootstrap starting.
I20260927 22:51:43.960372 5514 tablet_bootstrap.cc:654] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:43.961031 5513 tablet_bootstrap.cc:492] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8: No bootstrap required, opened a new log
I20260927 22:51:43.961236 5513 ts_tablet_manager.cc:1397] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:51:43.961544 5514 log.cc:824] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:43.962999 5515 tablet_bootstrap.cc:654] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:43.964382 5515 log.cc:824] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:43.967018 5515 tablet_bootstrap.cc:492] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: No bootstrap required, opened a new log
I20260927 22:51:43.967267 5515 ts_tablet_manager.cc:1397] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:51:43.967514 5514 tablet_bootstrap.cc:492] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: No bootstrap required, opened a new log
I20260927 22:51:43.967754 5514 ts_tablet_manager.cc:1397] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 22:51:43.970057 5513 raft_consensus.cc:348] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 } } peers { permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 } } peers { permanent_uuid: "37eaac87a19f4842ae411849f0c44cd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33849 } }
I20260927 22:51:43.970469 5513 raft_consensus.cc:374] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:43.970558 5513 raft_consensus.cc:729] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37eaac87a19f4842ae411849f0c44cd8, State: Initialized, Role: FOLLOWER
I20260927 22:51:43.971244 5513 consensus_queue.cc:260] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 } } peers { permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 } } peers { permanent_uuid: "37eaac87a19f4842ae411849f0c44cd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33849 } }
I20260927 22:51:43.972268 5235 heartbeater.cc:503] Master 127.0.97.62:40659 was elected leader, sending a full tablet report...
I20260927 22:51:43.972836 5513 ts_tablet_manager.cc:1428] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260927 22:51:43.975319 5515 raft_consensus.cc:348] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 } } peers { permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 } } peers { permanent_uuid: "37eaac87a19f4842ae411849f0c44cd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33849 } }
I20260927 22:51:43.975638 5515 raft_consensus.cc:374] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:43.975756 5515 raft_consensus.cc:729] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc869d5c57c5432385a1d6aa98f2d22f, State: Initialized, Role: FOLLOWER
I20260927 22:51:43.976297 5515 consensus_queue.cc:260] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 } } peers { permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 } } peers { permanent_uuid: "37eaac87a19f4842ae411849f0c44cd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33849 } }
I20260927 22:51:43.977319 5495 heartbeater.cc:503] Master 127.0.97.62:40659 was elected leader, sending a full tablet report...
I20260927 22:51:43.978006 5515 ts_tablet_manager.cc:1428] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 22:51:43.978034 5514 raft_consensus.cc:348] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 } } peers { permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 } } peers { permanent_uuid: "37eaac87a19f4842ae411849f0c44cd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33849 } }
I20260927 22:51:43.978575 5514 raft_consensus.cc:374] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:43.978681 5514 raft_consensus.cc:729] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce18e2f83e15415b96a6668505857f35, State: Initialized, Role: FOLLOWER
I20260927 22:51:43.979264 5514 consensus_queue.cc:260] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 } } peers { permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 } } peers { permanent_uuid: "37eaac87a19f4842ae411849f0c44cd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33849 } }
I20260927 22:51:43.980264 5365 heartbeater.cc:503] Master 127.0.97.62:40659 was elected leader, sending a full tablet report...
I20260927 22:51:43.980959 5514 ts_tablet_manager.cc:1428] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260927 22:51:44.038697 5236 tablet.cc:2369] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:44.092619 5519 raft_consensus.cc:482] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:44.092876 5519 raft_consensus.cc:504] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 } } peers { permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 } } peers { permanent_uuid: "37eaac87a19f4842ae411849f0c44cd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33849 } }
I20260927 22:51:44.094444 5519 leader_election.cc:288] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc869d5c57c5432385a1d6aa98f2d22f (127.0.97.3:40487), ce18e2f83e15415b96a6668505857f35 (127.0.97.2:34139)
I20260927 22:51:44.097749 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.094836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52017 (local address 127.0.97.3:40487)
0927 22:51:44.094989 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:44.094994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:44.095534 (+ 540us) server_negotiation.cc:407] Connection header received
0927 22:51:44.095710 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:44.095718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:44.095789 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:44.095889 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:44.097038 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:44.097044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:44.097049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:44.097067 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:44.097107 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:44.097125 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:44.097248 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:44.097381 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 22:51:44.097440 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:51:44.098016 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.095677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35339 (local address 127.0.97.2:34139)
0927 22:51:44.095880 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:44.095888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:44.096191 (+ 303us) server_negotiation.cc:407] Connection header received
0927 22:51:44.096362 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:44.096371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:44.096416 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:44.096510 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:44.097342 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:44.097358 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:44.097362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:44.097380 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:44.097412 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:44.097429 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:44.097557 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:44.097707 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 22:51:44.097768 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:51:44.098147 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.094859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40487 (local address 127.0.97.1:52017)
0927 22:51:44.095432 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.095447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:44.095588 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:44.095998 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:44.096003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:44.096030 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:44.096803 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:44.096805 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:44.096818 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:44.096876 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:44.097226 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:44.097229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:44.097233 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 22:51:44.097395 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 22:51:44.097440 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":283,"threads_started":1}
I20260927 22:51:44.098425 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.095512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34139 (local address 127.0.97.1:35339)
0927 22:51:44.096087 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.096103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:44.096231 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:44.096529 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:44.096534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:44.096556 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:44.097166 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:51:44.097171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:44.097188 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:51:44.097230 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:44.097561 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:44.097567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:44.097574 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:51:44.097746 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 22:51:44.097795 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":560,"threads_started":1}
I20260927 22:51:44.099066 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64002a8854cd44f7af093babbcc44709" candidate_uuid: "37eaac87a19f4842ae411849f0c44cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" is_pre_election: true
I20260927 22:51:44.099262 5320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64002a8854cd44f7af093babbcc44709" candidate_uuid: "37eaac87a19f4842ae411849f0c44cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce18e2f83e15415b96a6668505857f35" is_pre_election: true
I20260927 22:51:44.099565 5450 raft_consensus.cc:2454] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37eaac87a19f4842ae411849f0c44cd8 in term 0.
I20260927 22:51:44.099864 5320 raft_consensus.cc:2454] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37eaac87a19f4842ae411849f0c44cd8 in term 0.
I20260927 22:51:44.100203 5124 leader_election.cc:302] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37eaac87a19f4842ae411849f0c44cd8, dc869d5c57c5432385a1d6aa98f2d22f; no voters:
I20260927 22:51:44.100556 5519 raft_consensus.cc:2781] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:44.100690 5519 raft_consensus.cc:482] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:44.100796 5519 raft_consensus.cc:3037] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:44.102933 5519 raft_consensus.cc:504] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 } } peers { permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 } } peers { permanent_uuid: "37eaac87a19f4842ae411849f0c44cd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33849 } }
I20260927 22:51:44.103456 5519 leader_election.cc:288] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [CANDIDATE]: Term 1 election: Requested vote from peers dc869d5c57c5432385a1d6aa98f2d22f (127.0.97.3:40487), ce18e2f83e15415b96a6668505857f35 (127.0.97.2:34139)
I20260927 22:51:44.103814 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64002a8854cd44f7af093babbcc44709" candidate_uuid: "37eaac87a19f4842ae411849f0c44cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc869d5c57c5432385a1d6aa98f2d22f"
I20260927 22:51:44.103905 5320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64002a8854cd44f7af093babbcc44709" candidate_uuid: "37eaac87a19f4842ae411849f0c44cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce18e2f83e15415b96a6668505857f35"
I20260927 22:51:44.104017 5450 raft_consensus.cc:3037] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:44.104144 5320 raft_consensus.cc:3037] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:44.106453 5450 raft_consensus.cc:2454] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37eaac87a19f4842ae411849f0c44cd8 in term 1.
I20260927 22:51:44.106491 5320 raft_consensus.cc:2454] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37eaac87a19f4842ae411849f0c44cd8 in term 1.
I20260927 22:51:44.106783 5124 leader_election.cc:302] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37eaac87a19f4842ae411849f0c44cd8, dc869d5c57c5432385a1d6aa98f2d22f; no voters:
I20260927 22:51:44.107070 5519 raft_consensus.cc:2781] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:44.107484 5519 raft_consensus.cc:686] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [term 1 LEADER]: Becoming Leader. State: Replica: 37eaac87a19f4842ae411849f0c44cd8, State: Running, Role: LEADER
I20260927 22:51:44.107921 5519 consensus_queue.cc:237] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 } } peers { permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 } } peers { permanent_uuid: "37eaac87a19f4842ae411849f0c44cd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33849 } }
I20260927 22:51:44.112114 5049 catalog_manager.cc:5495] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37eaac87a19f4842ae411849f0c44cd8 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "37eaac87a19f4842ae411849f0c44cd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37eaac87a19f4842ae411849f0c44cd8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33849 } health_report { overall_health: HEALTHY } } }
I20260927 22:51:44.127761 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.125789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.0.97.1:33849)
0927 22:51:44.125966 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:44.125974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:44.125992 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:51:44.126214 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:44.126220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:44.126297 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:44.126400 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:44.127185 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:44.127190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:44.127195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:44.127210 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:44.127246 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:44.127257 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:44.127365 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:44.127460 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 22:51:44.127514 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:51:44.134307 5496 tablet.cc:2369] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:44.135326 5320 raft_consensus.cc:1261] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35 [term 1 FOLLOWER]: Refusing update from remote peer 37eaac87a19f4842ae411849f0c44cd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:44.135339 5450 raft_consensus.cc:1261] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f [term 1 FOLLOWER]: Refusing update from remote peer 37eaac87a19f4842ae411849f0c44cd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:44.136175 5524 consensus_queue.cc:1035] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce18e2f83e15415b96a6668505857f35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:44.136508 5519 consensus_queue.cc:1035] T 64002a8854cd44f7af093babbcc44709 P 37eaac87a19f4842ae411849f0c44cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc869d5c57c5432385a1d6aa98f2d22f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:44.146926 5527 mvcc.cc:205] Tried to move back new op lower bound from 7334090768928931840 to 7334090768829808640. Current Snapshot: MvccSnapshot[applied={T|T < 7334090768928931840}]
I20260927 22:51:44.152148 5528 mvcc.cc:205] Tried to move back new op lower bound from 7334090768928931840 to 7334090768829808640. Current Snapshot: MvccSnapshot[applied={T|T < 7334090768928931840}]
I20260927 22:51:44.164184 5049 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:44.164428 5049 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:44.167476 5049 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b407de795a14df08603b26540e7a896: Sending DeleteTablet for 3 replicas of tablet 64002a8854cd44f7af093babbcc44709
I20260927 22:51:44.168533 5300 tablet_service.cc:1514] Processing DeleteTablet for tablet 64002a8854cd44f7af093babbcc44709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:44 UTC) from {username='slave'} at 127.0.0.1:59730
I20260927 22:51:44.168571 5170 tablet_service.cc:1514] Processing DeleteTablet for tablet 64002a8854cd44f7af093babbcc44709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:44 UTC) from {username='slave'} at 127.0.0.1:34582
I20260927 22:51:44.169488 5430 tablet_service.cc:1514] Processing DeleteTablet for tablet 64002a8854cd44f7af093babbcc44709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:44 UTC) from {username='slave'} at 127.0.0.1:37252
I20260927 22:51:44.170987 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 5108
I20260927 22:51:44.180063 5539 tablet_replica.cc:330] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: stopping tablet replica
I20260927 22:51:44.180591 5539 raft_consensus.cc:2229] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:44.181162 5537 tablet_replica.cc:330] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: stopping tablet replica
I20260927 22:51:44.181156 5539 raft_consensus.cc:2258] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:44.181645 5537 raft_consensus.cc:2229] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:44.182209 5537 raft_consensus.cc:2258] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:44.184119 5537 ts_tablet_manager.cc:1905] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:44.186503 5539 ts_tablet_manager.cc:1905] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:44.188583 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.172645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.0.97.1:33849)
0927 22:51:44.172850 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:44.172859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:44.186084 (+ 13225us) server_negotiation.cc:407] Connection header received
0927 22:51:44.186562 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:44.186569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:44.186633 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:44.186703 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:44.187830 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:44.187835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:44.187840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:44.187858 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:44.187890 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:44.187904 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:44.187991 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:44.188236 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 22:51:44.188293 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:51:44.189560 5230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:44.194334 5537 ts_tablet_manager.cc:1918] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:44.194547 5539 ts_tablet_manager.cc:1918] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:44.194585 5537 log.cc:1196] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/64002a8854cd44f7af093babbcc44709
I20260927 22:51:44.194742 5539 log.cc:1196] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/64002a8854cd44f7af093babbcc44709
I20260927 22:51:44.195375 5537 ts_tablet_manager.cc:1939] T 64002a8854cd44f7af093babbcc44709 P ce18e2f83e15415b96a6668505857f35: Deleting consensus metadata
I20260927 22:51:44.195380 5539 ts_tablet_manager.cc:1939] T 64002a8854cd44f7af093babbcc44709 P dc869d5c57c5432385a1d6aa98f2d22f: Deleting consensus metadata
I20260927 22:51:44.197556 5034 catalog_manager.cc:4841] TS dc869d5c57c5432385a1d6aa98f2d22f (127.0.97.3:40487): tablet 64002a8854cd44f7af093babbcc44709 (table test-table [id=0b482ef7dade444090e007ffdd5094f2]) successfully deleted
I20260927 22:51:44.197914 5034 catalog_manager.cc:4841] TS ce18e2f83e15415b96a6668505857f35 (127.0.97.2:34139): tablet 64002a8854cd44f7af093babbcc44709 (table test-table [id=0b482ef7dade444090e007ffdd5094f2]) successfully deleted
I20260927 22:51:44.285775 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 5108
W20260927 22:51:44.303234 5035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33849 (error 108)
I20260927 22:51:44.303448 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 5238
I20260927 22:51:44.305060 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.304598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33849 (local address 127.0.0.1:34598)
0927 22:51:44.304742 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.304898 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 22:51:44.305274 5035 catalog_manager.cc:4584] TS 37eaac87a19f4842ae411849f0c44cd8 (127.0.97.1:33849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64002a8854cd44f7af093babbcc44709: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33849: connect: Connection refused (error 111)
I20260927 22:51:44.306334 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.304137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.0.97.2:34139)
0927 22:51:44.304313 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:44.304324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:44.304428 (+ 104us) server_negotiation.cc:407] Connection header received
0927 22:51:44.304606 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:44.304612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:44.304718 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:44.304816 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:44.305763 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:44.305771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:44.305779 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:44.305799 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:44.305838 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:44.305854 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:44.305982 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:44.306016 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 22:51:44.306056 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:51:44.306906 5360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:44.358946 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.358510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33849 (local address 127.0.0.1:34606)
0927 22:51:44.358694 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.358775 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:51:44.359860 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.359566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33849 (local address 127.0.0.1:34610)
0927 22:51:44.359685 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.359756 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 22:51:44.392336 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 5238
I20260927 22:51:44.410606 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 5368
I20260927 22:51:44.413261 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.411315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.0.97.3:40487)
0927 22:51:44.411488 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:44.411496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:44.411516 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:51:44.411708 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:44.411713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:44.411801 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:44.411905 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:44.412653 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:44.412659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:44.412663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:44.412680 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:44.412717 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:44.412728 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:44.412854 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:44.412928 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:51:44.412985 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:51:44.413858 5490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:44.420150 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.419617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33849 (local address 127.0.0.1:34622)
0927 22:51:44.419863 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.419961 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 22:51:44.421180 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.420749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33849 (local address 127.0.0.1:34638)
0927 22:51:44.420919 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.421006 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 22:51:44.499155 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 5368
I20260927 22:51:44.501921 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.501412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33849 (local address 127.0.0.1:34646)
0927 22:51:44.501685 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.501784 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 22:51:44.503021 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:44.502649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33849 (local address 127.0.0.1:34652)
0927 22:51:44.502845 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:44.502918 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 22:51:44.517020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 5018
I20260927 22:51:44.518077 5079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:44.621258 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 5018
2026-09-27T22:51:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2535 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 22:51:44.657015 388 mini_oidc.cc:152] Starting JWKS server
I20260927 22:51:44.659514 388 webserver.cc:460] Webserver started at https://127.0.0.1:34571/ using document root <none> and password file <none>
I20260927 22:51:44.659814 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:51:44.660384 388 webserver.cc:460] Webserver started at http://127.0.0.1:46483/ using document root <none> and password file <none>
2026-09-27T22:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:44Z Disabled control of system clock
I20260927 22:51:44.683959 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45727
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45727
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34571/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/testchainca.cert with env {}
W20260927 22:51:44.983255 5551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:44.983513 5551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:44.983671 5551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:44.990657 5551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:44.990748 5551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:44.990834 5551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:44.990911 5551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:44.990973 5551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:51:44.991068 5551 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:51:44.991133 5551 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34571/jwks.json
I20260927 22:51:45.000540 5551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34571/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:45.001704 5551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:45.002882 5551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:45.014051 5556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:45.014086 5557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:45.014169 5559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:45.014940 5551 server_base.cc:1034] running on GCE node
I20260927 22:51:45.015548 5551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:45.016727 5551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:45.017944 5551 hybrid_clock.cc:648] HybridClock initialized: now 1790549505017905 us; error 30 us; skew 500 ppm
I20260927 22:51:45.018251 5551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:45.026155 5551 webserver.cc:460] Webserver started at http://127.0.97.62:37893/ using document root <none> and password file <none>
I20260927 22:51:45.026777 5551 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:45.026906 5551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:45.027184 5551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:45.029855 5551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "ef63d9bb71784fde9c06e92ebfd6be7a"
format_stamp: "Formatted at 2026-09-27 22:51:45 on dist-test-slave-df53"
server_key: "c60a100ffefb9065ead7434ce2bf5413"
server_key_iv: "ed4d2b24f5aa10e42372f9bbaeda8f9a"
server_key_version: "encryptionkey@0"
I20260927 22:51:45.030586 5551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef63d9bb71784fde9c06e92ebfd6be7a"
format_stamp: "Formatted at 2026-09-27 22:51:45 on dist-test-slave-df53"
server_key: "c60a100ffefb9065ead7434ce2bf5413"
server_key_iv: "ed4d2b24f5aa10e42372f9bbaeda8f9a"
server_key_version: "encryptionkey@0"
I20260927 22:51:45.035229 5551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:45.038331 5565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:45.039487 5551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:51:45.039659 5551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "ef63d9bb71784fde9c06e92ebfd6be7a"
format_stamp: "Formatted at 2026-09-27 22:51:45 on dist-test-slave-df53"
server_key: "c60a100ffefb9065ead7434ce2bf5413"
server_key_iv: "ed4d2b24f5aa10e42372f9bbaeda8f9a"
server_key_version: "encryptionkey@0"
I20260927 22:51:45.039861 5551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:45.053913 5551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:45.054602 5551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:45.054860 5551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:45.071949 5551 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45727
I20260927 22:51:45.071978 5616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45727 every 8 connection(s)
I20260927 22:51:45.073134 5551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:45.076431 5617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:45.077950 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 5551
I20260927 22:51:45.078168 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:45.078944 388 external_mini_cluster.cc:1428] Setting key ec203a25d4d1ba4fc0fd6966c8957e39
I20260927 22:51:45.088950 5617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a: Bootstrap starting.
I20260927 22:51:45.091600 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.080736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.0.97.62:45727)
0927 22:51:45.081772 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.081782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.081826 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:51:45.083576 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.083598 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.084014 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.084420 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:45.084945 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.086251 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.087549 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.088158 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.088347 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.089499 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:45.089505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:45.089517 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:45.090069 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:45.090143 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:45.090162 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:45.090298 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.090505 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.090761 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":319,"threads_started":1}
I20260927 22:51:45.091998 5617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:45.092949 5617 log.cc:824] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:45.095048 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:51:45.095722 5617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a: No bootstrap required, opened a new log
I20260927 22:51:45.103006 5617 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d9bb71784fde9c06e92ebfd6be7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45727 } }
I20260927 22:51:45.103407 5617 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:45.103516 5617 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef63d9bb71784fde9c06e92ebfd6be7a, State: Initialized, Role: FOLLOWER
I20260927 22:51:45.104019 5617 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d9bb71784fde9c06e92ebfd6be7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45727 } }
I20260927 22:51:45.104238 5617 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:45.104367 5617 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:45.104527 5617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:45.106544 5617 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d9bb71784fde9c06e92ebfd6be7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45727 } }
I20260927 22:51:45.106982 5617 leader_election.cc:302] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef63d9bb71784fde9c06e92ebfd6be7a; no voters:
I20260927 22:51:45.107501 5617 leader_election.cc:288] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:45.107585 5622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:45.108212 5622 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [term 1 LEADER]: Becoming Leader. State: Replica: ef63d9bb71784fde9c06e92ebfd6be7a, State: Running, Role: LEADER
I20260927 22:51:45.108726 5622 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d9bb71784fde9c06e92ebfd6be7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45727 } }
I20260927 22:51:45.109498 5617 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:45.111600 5623 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef63d9bb71784fde9c06e92ebfd6be7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d9bb71784fde9c06e92ebfd6be7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45727 } } }
I20260927 22:51:45.112020 5623 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:45.111845 5624 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef63d9bb71784fde9c06e92ebfd6be7a. Latest consensus state: current_term: 1 leader_uuid: "ef63d9bb71784fde9c06e92ebfd6be7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63d9bb71784fde9c06e92ebfd6be7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45727 } } }
I20260927 22:51:45.112510 5624 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:45.113953 5631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:45.118201 5631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:45.125224 5631 catalog_manager.cc:1347] Generated new cluster ID: 9efc8107b5084109afdae9b9861e31b6
I20260927 22:51:45.125322 5631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:45.142081 5631 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:45.143417 5631 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:45.155339 5631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef63d9bb71784fde9c06e92ebfd6be7a: Generated new TSK 0
I20260927 22:51:45.156090 5631 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:51:45.199442 5647 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 22:51:45.199581 5647 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:51:45.200659 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.179786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.0.97.62:45727)
0927 22:51:45.179963 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.179970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.180480 (+ 510us) server_negotiation.cc:407] Connection header received
0927 22:51:45.180719 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.180726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.180813 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.180951 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:51:45.181572 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.182690 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.183987 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.184677 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.184805 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.185221 (+ 416us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:51:45.200052 (+ 14831us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:51:45.200367 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34340: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 22:51:45.200771 5620 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34340: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 22:51:45.201977 5646 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:51:45.179732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45727 (local address 127.0.0.1:34340)
0927 22:51:45.180392 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:45.180407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:45.180539 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:45.180982 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:45.180985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:45.180994 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:51:45.181408 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.181423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.182811 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.182815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.183864 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.183873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.184790 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.184836 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.184946 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.184992 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:51:45.200313 (+ 15321us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:51:45.200529 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45727: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":350,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:45727: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34571/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 22:51:45.203580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 5551
I20260927 22:51:45.204573 5612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:45.295457 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 5551
2026-09-27T22:51:45Z chronyd exiting
I20260927 22:51:45.344656 388 test_util.cc:175] -----------------------------------------------
I20260927 22:51:45.344820 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549465302610-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (691 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 22:51:45.348394 388 mini_oidc.cc:152] Starting JWKS server
I20260927 22:51:45.350200 388 webserver.cc:460] Webserver started at https://127.0.0.1:38687/ using document root <none> and password file <none>
I20260927 22:51:45.350414 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:51:45.351027 388 webserver.cc:460] Webserver started at http://127.0.0.1:45497/ using document root <none> and password file <none>
2026-09-27T22:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:45Z Disabled control of system clock
I20260927 22:51:45.375150 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42719
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42719
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38687/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/testchainca.cert with env {}
W20260927 22:51:45.688691 5654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:45.688947 5654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:45.689031 5654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:45.696806 5654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:45.696909 5654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:45.696985 5654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:45.697074 5654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:45.697146 5654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:51:45.697230 5654 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:51:45.697299 5654 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38687/jwks.json
I20260927 22:51:45.708969 5654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38687/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:45.710320 5654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:45.711556 5654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:45.723057 5659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:45.723132 5660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:45.723135 5662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:45.723763 5654 server_base.cc:1034] running on GCE node
I20260927 22:51:45.724540 5654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:45.725895 5654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:45.727103 5654 hybrid_clock.cc:648] HybridClock initialized: now 1790549505727073 us; error 45 us; skew 500 ppm
I20260927 22:51:45.727432 5654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:45.729533 5654 webserver.cc:460] Webserver started at http://127.0.97.62:44201/ using document root <none> and password file <none>
I20260927 22:51:45.730212 5654 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:45.730342 5654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:45.730649 5654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:45.733412 5654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "3ba293011f484ffd9c4c1b3f7f6befb8"
format_stamp: "Formatted at 2026-09-27 22:51:45 on dist-test-slave-df53"
server_key: "a79626d2a368b8a1ab164f6c95dbe3d8"
server_key_iv: "2455b3337978280a7bbf3747266870db"
server_key_version: "encryptionkey@0"
I20260927 22:51:45.734216 5654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "3ba293011f484ffd9c4c1b3f7f6befb8"
format_stamp: "Formatted at 2026-09-27 22:51:45 on dist-test-slave-df53"
server_key: "a79626d2a368b8a1ab164f6c95dbe3d8"
server_key_iv: "2455b3337978280a7bbf3747266870db"
server_key_version: "encryptionkey@0"
I20260927 22:51:45.738876 5654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 22:51:45.742064 5668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:45.743306 5654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:51:45.743479 5654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "3ba293011f484ffd9c4c1b3f7f6befb8"
format_stamp: "Formatted at 2026-09-27 22:51:45 on dist-test-slave-df53"
server_key: "a79626d2a368b8a1ab164f6c95dbe3d8"
server_key_iv: "2455b3337978280a7bbf3747266870db"
server_key_version: "encryptionkey@0"
I20260927 22:51:45.743677 5654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:45.775995 5654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:45.776832 5654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:45.777112 5654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:45.794672 5654 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42719
I20260927 22:51:45.794680 5719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42719 every 8 connection(s)
I20260927 22:51:45.795948 5654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:45.799499 5720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:45.804113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 5654
I20260927 22:51:45.804307 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:45.804934 388 external_mini_cluster.cc:1428] Setting key 8dbc0cf88942928b813c6546bff1c9f2
I20260927 22:51:45.809990 5720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8: Bootstrap starting.
I20260927 22:51:45.812937 5720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:45.813997 5720 log.cc:824] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:45.816710 5720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8: No bootstrap required, opened a new log
I20260927 22:51:45.817487 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.806726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.0.97.62:42719)
0927 22:51:45.807710 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.807724 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.807762 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:51:45.809334 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.809355 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.809849 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.810210 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:45.810738 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.811977 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.813227 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.814010 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.814203 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.815308 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:45.815321 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:45.815334 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:45.815826 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:45.815900 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:45.815919 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:45.816041 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:45.816253 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 22:51:45.816527 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":365,"threads_started":1}
I20260927 22:51:45.821033 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:51:45.824769 5720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba293011f484ffd9c4c1b3f7f6befb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42719 } }
I20260927 22:51:45.825110 5720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:45.825207 5720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba293011f484ffd9c4c1b3f7f6befb8, State: Initialized, Role: FOLLOWER
I20260927 22:51:45.825837 5720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba293011f484ffd9c4c1b3f7f6befb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42719 } }
I20260927 22:51:45.826105 5720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:45.826241 5720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:45.826448 5720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:45.828461 5720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba293011f484ffd9c4c1b3f7f6befb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42719 } }
I20260927 22:51:45.828913 5720 leader_election.cc:302] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ba293011f484ffd9c4c1b3f7f6befb8; no voters:
I20260927 22:51:45.829489 5720 leader_election.cc:288] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:45.829581 5725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:45.830339 5725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba293011f484ffd9c4c1b3f7f6befb8, State: Running, Role: LEADER
I20260927 22:51:45.830834 5725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba293011f484ffd9c4c1b3f7f6befb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42719 } }
I20260927 22:51:45.831472 5720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:45.833532 5727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba293011f484ffd9c4c1b3f7f6befb8. Latest consensus state: current_term: 1 leader_uuid: "3ba293011f484ffd9c4c1b3f7f6befb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba293011f484ffd9c4c1b3f7f6befb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42719 } } }
I20260927 22:51:45.833931 5727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:45.833654 5726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ba293011f484ffd9c4c1b3f7f6befb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba293011f484ffd9c4c1b3f7f6befb8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42719 } } }
I20260927 22:51:45.834316 5726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:45.835651 5731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:45.840771 5731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:45.849262 5731 catalog_manager.cc:1347] Generated new cluster ID: a95e1979b0514911aeef87b36ecf1301
I20260927 22:51:45.849373 5731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:45.877759 5731 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:45.879155 5731 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:45.888764 5731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ba293011f484ffd9c4c1b3f7f6befb8: Generated new TSK 0
I20260927 22:51:45.889568 5731 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:51:45.935427 5750 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 22:51:45.935552 5750 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:51:45.936597 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:45.909028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.0.97.62:42719)
0927 22:51:45.909236 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:45.909246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:45.909731 (+ 485us) server_negotiation.cc:407] Connection header received
0927 22:51:45.909966 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:45.909971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:45.910035 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:45.910158 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:51:45.910683 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.911381 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.913404 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.913894 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.913999 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.914508 (+ 509us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:51:45.935960 (+ 21452us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38687/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:51:45.936343 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53604: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38687/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:51:45.936741 5723 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53604: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38687/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 22:51:45.936915 5749 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:51:45.909005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42719 (local address 127.0.0.1:53604)
0927 22:51:45.909630 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:45.909654 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:45.909797 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:45.910209 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:45.910214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:45.910229 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:51:45.910543 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.910554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.911511 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.911517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.913243 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:45.913258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:45.914012 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:45.914057 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:45.914244 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:45.914287 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:51:45.936247 (+ 21960us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38687/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:51:45.936531 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42719: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38687/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":321,"threads_started":1}
I20260927 22:51:45.938241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 5654
I20260927 22:51:45.939402 5715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:46.031419 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 5654
2026-09-27T22:51:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (738 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 22:51:46.087617 388 mini_oidc.cc:152] Starting JWKS server
I20260927 22:51:46.089346 388 webserver.cc:460] Webserver started at https://127.0.0.1:46485/ using document root <none> and password file <none>
I20260927 22:51:46.089545 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:51:46.090162 388 webserver.cc:460] Webserver started at http://127.0.0.1:40831/ using document root <none> and password file <none>
2026-09-27T22:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:46Z Disabled control of system clock
I20260927 22:51:46.114768 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33843
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33843
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46485/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:51:46.421928 5757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:46.422269 5757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:46.422425 5757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:46.429396 5757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:46.429525 5757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:46.429654 5757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:46.429751 5757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:46.429812 5757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:51:46.429893 5757 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:51:46.429960 5757 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46485/jwks.json
I20260927 22:51:46.439149 5757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46485/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:46.440399 5757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:46.442216 5757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:46.453996 5762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:46.454066 5763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:46.454082 5765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:46.459235 5757 server_base.cc:1034] running on GCE node
I20260927 22:51:46.460021 5757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:46.461386 5757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:46.462599 5757 hybrid_clock.cc:648] HybridClock initialized: now 1790549506462561 us; error 34 us; skew 500 ppm
I20260927 22:51:46.462936 5757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:46.465243 5757 webserver.cc:460] Webserver started at http://127.0.97.62:44337/ using document root <none> and password file <none>
I20260927 22:51:46.465986 5757 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:46.466105 5757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:46.466421 5757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:46.469156 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7"
format_stamp: "Formatted at 2026-09-27 22:51:46 on dist-test-slave-df53"
server_key: "d2cb15a406393d7039181aae6baec201"
server_key_iv: "5891d93e8000e7bcbc8e79728b225cd5"
server_key_version: "encryptionkey@0"
I20260927 22:51:46.469966 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7"
format_stamp: "Formatted at 2026-09-27 22:51:46 on dist-test-slave-df53"
server_key: "d2cb15a406393d7039181aae6baec201"
server_key_iv: "5891d93e8000e7bcbc8e79728b225cd5"
server_key_version: "encryptionkey@0"
I20260927 22:51:46.474974 5757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:51:46.478135 5771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:46.479399 5757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:51:46.479573 5757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7"
format_stamp: "Formatted at 2026-09-27 22:51:46 on dist-test-slave-df53"
server_key: "d2cb15a406393d7039181aae6baec201"
server_key_iv: "5891d93e8000e7bcbc8e79728b225cd5"
server_key_version: "encryptionkey@0"
I20260927 22:51:46.479777 5757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:46.504936 5757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:46.505831 5757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:46.506108 5757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:46.523912 5757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33843
I20260927 22:51:46.523934 5822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33843 every 8 connection(s)
I20260927 22:51:46.525213 5757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:46.528692 5823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:46.530865 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 5757
I20260927 22:51:46.531078 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:46.531857 388 external_mini_cluster.cc:1428] Setting key f8e13f8e2c13175a133230844184e82b
I20260927 22:51:46.539958 5823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7: Bootstrap starting.
I20260927 22:51:46.542840 5823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:46.543859 5823 log.cc:824] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:46.544483 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:46.533851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.0.97.62:33843)
0927 22:51:46.534816 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:46.534825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:46.534866 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:51:46.536405 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:46.536425 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:46.536852 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:46.537241 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:46.537792 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.539019 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.540304 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.540933 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.541115 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:46.542226 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:46.542233 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:46.542246 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:46.542722 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:46.542829 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:46.542848 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:46.542969 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:46.543204 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 22:51:46.543467 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":367,"threads_started":1}
I20260927 22:51:46.546799 5823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7: No bootstrap required, opened a new log
I20260927 22:51:46.547953 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:51:46.554160 5823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33843 } }
I20260927 22:51:46.554484 5823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:46.554584 5823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5f89a5fdce4d4e85b6bb033513fcc7, State: Initialized, Role: FOLLOWER
I20260927 22:51:46.555138 5823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33843 } }
I20260927 22:51:46.555352 5823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:46.555500 5823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:46.555686 5823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:46.557812 5823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33843 } }
I20260927 22:51:46.558260 5823 leader_election.cc:302] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b5f89a5fdce4d4e85b6bb033513fcc7; no voters:
I20260927 22:51:46.558835 5823 leader_election.cc:288] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:46.559078 5828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:46.559760 5828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [term 1 LEADER]: Becoming Leader. State: Replica: 6b5f89a5fdce4d4e85b6bb033513fcc7, State: Running, Role: LEADER
I20260927 22:51:46.560236 5828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33843 } }
I20260927 22:51:46.560748 5823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:46.562909 5830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b5f89a5fdce4d4e85b6bb033513fcc7. Latest consensus state: current_term: 1 leader_uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33843 } } }
I20260927 22:51:46.563105 5829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b5f89a5fdce4d4e85b6bb033513fcc7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33843 } } }
I20260927 22:51:46.563345 5830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:46.563480 5829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:46.565635 5837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:46.570585 5837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:46.578745 5837 catalog_manager.cc:1347] Generated new cluster ID: 1bf705f11b804fb38ff80f7a86db0d5f
I20260927 22:51:46.578863 5837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:46.620684 5837 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:46.622287 5837 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:46.628082 5837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b5f89a5fdce4d4e85b6bb033513fcc7: Generated new TSK 0
I20260927 22:51:46.628822 5837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:51:46.654629 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:46.646880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.0.97.62:33843)
0927 22:51:46.647061 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:46.647068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:46.647556 (+ 488us) server_negotiation.cc:407] Connection header received
0927 22:51:46.647746 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:46.647751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:46.647836 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:46.647981 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:51:46.648570 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.649753 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.651072 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.651585 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.651738 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:46.652192 (+ 454us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:51:46.654007 (+ 1815us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46485/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:51:46.654314 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46485/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 22:51:46.654873 5852 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:51:46.646874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33843 (local address 127.0.0.1:40818)
0927 22:51:46.647467 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:46.647482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:46.647594 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:46.648031 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:46.648038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:46.648053 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:51:46.648415 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:46.648427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.649890 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.649900 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:46.650934 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:46.650945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:46.651718 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:46.651805 (+ 87us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:46.651945 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:46.651974 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:51:46.654184 (+ 2210us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46485/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:51:46.654406 (+ 222us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46485/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":327,"threads_started":1}
W20260927 22:51:46.655093 5826 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46485/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 22:51:46.655620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 5757
I20260927 22:51:46.656627 5818 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 22:51:46.666772 5853 webserver.cc:533] Webserver: sslize failed:
E20260927 22:51:46.666882 5853 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:51:46.745886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 5757
2026-09-27T22:51:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (709 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T22:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:46Z Disabled control of system clock
I20260927 22:51:46.820768 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39211
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39211
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/testchainca.cert with env {}
W20260927 22:51:47.128732 5858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:47.129004 5858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:47.129151 5858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:47.136369 5858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:47.136483 5858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:47.136564 5858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:47.136656 5858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:47.136742 5858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:51:47.136839 5858 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:51:47.136894 5858 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 22:51:47.146708 5858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:47.147900 5858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:47.149106 5858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:47.160877 5863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:47.160955 5866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:47.160969 5864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:47.161828 5858 server_base.cc:1034] running on GCE node
I20260927 22:51:47.162544 5858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:47.163901 5858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:47.165120 5858 hybrid_clock.cc:648] HybridClock initialized: now 1790549507165083 us; error 50 us; skew 500 ppm
I20260927 22:51:47.165467 5858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:47.167618 5858 webserver.cc:460] Webserver started at http://127.0.97.62:46329/ using document root <none> and password file <none>
I20260927 22:51:47.168279 5858 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:47.168437 5858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:47.168743 5858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:47.171584 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "68f0af6f2f23454f8d6b9435493bc191"
format_stamp: "Formatted at 2026-09-27 22:51:47 on dist-test-slave-df53"
server_key: "62c591f6408fb1be75f717b04a3c774f"
server_key_iv: "878216038062e3763f68c04e5486320e"
server_key_version: "encryptionkey@0"
I20260927 22:51:47.172356 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "68f0af6f2f23454f8d6b9435493bc191"
format_stamp: "Formatted at 2026-09-27 22:51:47 on dist-test-slave-df53"
server_key: "62c591f6408fb1be75f717b04a3c774f"
server_key_iv: "878216038062e3763f68c04e5486320e"
server_key_version: "encryptionkey@0"
I20260927 22:51:47.177202 5858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 22:51:47.180577 5872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:47.181843 5858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:47.182011 5858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "68f0af6f2f23454f8d6b9435493bc191"
format_stamp: "Formatted at 2026-09-27 22:51:47 on dist-test-slave-df53"
server_key: "62c591f6408fb1be75f717b04a3c774f"
server_key_iv: "878216038062e3763f68c04e5486320e"
server_key_version: "encryptionkey@0"
I20260927 22:51:47.182219 5858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:47.210606 5858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:47.211445 5858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:47.211730 5858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:47.229099 5858 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39211
I20260927 22:51:47.229107 5923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39211 every 8 connection(s)
I20260927 22:51:47.230391 5858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:47.234012 5924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:47.236835 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 5858
I20260927 22:51:47.237033 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:47.237708 388 external_mini_cluster.cc:1428] Setting key 48efbbdc6aa59b945fdd3d9a60165d65
I20260927 22:51:47.245855 5924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191: Bootstrap starting.
I20260927 22:51:47.249301 5924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:47.250402 5924 log.cc:824] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:47.250679 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.239549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.0.97.62:39211)
0927 22:51:47.240466 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:47.240476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:47.240515 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:51:47.242101 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:47.242122 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:47.242573 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:47.242887 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:47.243471 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.244701 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.246484 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.247108 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.247247 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.248465 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:47.248472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:47.248484 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:51:47.248974 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:47.249059 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:51:47.249079 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:47.249169 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:47.249435 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 22:51:47.249774 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":319,"threads_started":1}
I20260927 22:51:47.254246 5924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191: No bootstrap required, opened a new log
I20260927 22:51:47.254279 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:51:47.261986 5924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f0af6f2f23454f8d6b9435493bc191" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39211 } }
I20260927 22:51:47.262323 5924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:47.262431 5924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f0af6f2f23454f8d6b9435493bc191, State: Initialized, Role: FOLLOWER
I20260927 22:51:47.262941 5924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f0af6f2f23454f8d6b9435493bc191" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39211 } }
I20260927 22:51:47.263175 5924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:47.263312 5924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:47.263465 5924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:47.265938 5924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f0af6f2f23454f8d6b9435493bc191" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39211 } }
I20260927 22:51:47.266422 5924 leader_election.cc:302] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f0af6f2f23454f8d6b9435493bc191; no voters:
I20260927 22:51:47.266955 5924 leader_election.cc:288] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:47.267091 5929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:47.267783 5929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [term 1 LEADER]: Becoming Leader. State: Replica: 68f0af6f2f23454f8d6b9435493bc191, State: Running, Role: LEADER
I20260927 22:51:47.268271 5929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f0af6f2f23454f8d6b9435493bc191" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39211 } }
I20260927 22:51:47.268872 5924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:47.271085 5930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f0af6f2f23454f8d6b9435493bc191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f0af6f2f23454f8d6b9435493bc191" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39211 } } }
I20260927 22:51:47.271474 5930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:47.271446 5931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f0af6f2f23454f8d6b9435493bc191. Latest consensus state: current_term: 1 leader_uuid: "68f0af6f2f23454f8d6b9435493bc191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f0af6f2f23454f8d6b9435493bc191" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39211 } } }
I20260927 22:51:47.271840 5931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:47.272617 5938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:47.278239 5938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:47.286586 5938 catalog_manager.cc:1347] Generated new cluster ID: 0195d60120b94aa388f242b3b09e4e21
I20260927 22:51:47.286720 5938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:47.317337 5938 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:47.318652 5938 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:47.329085 5938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f0af6f2f23454f8d6b9435493bc191: Generated new TSK 0
I20260927 22:51:47.329931 5938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:51:47.858399 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:47.351617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.0.97.62:39211)
0927 22:51:47.351843 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:47.351851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:47.352100 (+ 249us) server_negotiation.cc:407] Connection header received
0927 22:51:47.352357 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:47.352361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:47.352436 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:47.352585 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:51:47.353029 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.353806 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.355196 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.355634 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.355808 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.356245 (+ 437us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:51:47.857723 (+501478us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:51:47.858104 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43714: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 22:51:47.858529 5927 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43714: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 22:51:47.858631 5953 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:51:47.351469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39211 (local address 127.0.0.1:43714)
0927 22:51:47.351994 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:47.352012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:47.352175 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:47.352599 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:47.352604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:47.352616 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:51:47.352891 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.352899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.353938 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.353944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.355051 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:47.355062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:47.355804 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:47.355855 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:47.355989 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:47.356021 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:51:47.857980 (+501959us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:51:47.858255 (+ 275us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":391,"threads_started":1}
I20260927 22:51:47.859750 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 5858
I20260927 22:51:47.860817 5919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:47.959389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 5858
2026-09-27T22:51:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1198 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (390 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (391 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:48 dist-test-slave-df53 krb5kdc[5968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:48 dist-test-slave-df53 krb5kdc[5968](info): set up 2 sockets
Sep 27 22:51:48 dist-test-slave-df53 krb5kdc[5968](info): commencing operation
krb5kdc: starting...
W20260927 22:51:49.814567 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:49 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:51:50.003021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35721
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:50.310532 5981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:50.310792 5981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:50.310930 5981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:50.317994 5981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:50.318080 5981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:50.318166 5981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:50.318257 5981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:50.318331 5981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:50.327672 5981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:50.328806 5981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:50.330018 5981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:50.341449 5986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:50.341416 5987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:50.341614 5989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:50.342165 5981 server_base.cc:1034] running on GCE node
I20260927 22:51:50.342861 5981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:50.344135 5981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:50.345366 5981 hybrid_clock.cc:648] HybridClock initialized: now 1790549510345323 us; error 52 us; skew 500 ppm
I20260927 22:51:50.345734 5981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:50 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:50.351625 5981 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:51:50.352897 5981 webserver.cc:460] Webserver started at http://127.0.97.62:35167/ using document root <none> and password file <none>
I20260927 22:51:50.353472 5981 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:50.353560 5981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:50.353852 5981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:50.356549 5981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "87ff88918faf4016956388144adb6ee0"
format_stamp: "Formatted at 2026-09-27 22:51:50 on dist-test-slave-df53"
server_key: "e0fddb442ca34bbaf0a63a1e4dd4ab7e"
server_key_iv: "d418d13fa05a4c7a34e93183d6eec854"
server_key_version: "encryptionkey@0"
I20260927 22:51:50.357234 5981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "87ff88918faf4016956388144adb6ee0"
format_stamp: "Formatted at 2026-09-27 22:51:50 on dist-test-slave-df53"
server_key: "e0fddb442ca34bbaf0a63a1e4dd4ab7e"
server_key_iv: "d418d13fa05a4c7a34e93183d6eec854"
server_key_version: "encryptionkey@0"
I20260927 22:51:50.361814 5981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260927 22:51:50.364913 5996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:50.366154 5981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:50.366292 5981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "87ff88918faf4016956388144adb6ee0"
format_stamp: "Formatted at 2026-09-27 22:51:50 on dist-test-slave-df53"
server_key: "e0fddb442ca34bbaf0a63a1e4dd4ab7e"
server_key_iv: "d418d13fa05a4c7a34e93183d6eec854"
server_key_version: "encryptionkey@0"
I20260927 22:51:50.366442 5981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:50.379202 5981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:50.383194 5981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:50.383421 5981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:50.400755 5981 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35721
I20260927 22:51:50.400787 6047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35721 every 8 connection(s)
I20260927 22:51:50.402100 5981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:50.405566 6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:50.409233 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 5981
I20260927 22:51:50.409436 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:50.410235 388 external_mini_cluster.cc:1428] Setting key cad7f16e06896190da8c103467fe8154
I20260927 22:51:50.418457 6048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0: Bootstrap starting.
I20260927 22:51:50.422341 6048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:50.423237 6048 log.cc:824] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:51:50 dist-test-slave-df53 krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:50.425789 6048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0: No bootstrap required, opened a new log
I20260927 22:51:50.432017 6051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:50.412192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.0.97.62:35721)
0927 22:51:50.412912 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:50.412922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:50.412961 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:51:50.414555 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:50.414575 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:50.414939 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:50.415236 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:50.416882 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:50.418155 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:50.419858 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:50.420955 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:50.421132 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:50.425217 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:50.425225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:50.425239 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:50.425271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:50.428312 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:50.428948 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:50.428955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:50.428960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:50.429124 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:50.429501 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:50.429506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:50.429516 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:50.430047 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:50.430343 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:50.430717 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 22:51:50.431089 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":341,"threads_started":1}
I20260927 22:51:50.433378 6048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87ff88918faf4016956388144adb6ee0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35721 } }
I20260927 22:51:50.433779 6048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:50.433892 6048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ff88918faf4016956388144adb6ee0, State: Initialized, Role: FOLLOWER
I20260927 22:51:50.434418 6048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87ff88918faf4016956388144adb6ee0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35721 } }
I20260927 22:51:50.434656 6048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:50.434788 6048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:50.434937 6048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:50.437090 6048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87ff88918faf4016956388144adb6ee0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35721 } }
I20260927 22:51:50.437512 6048 leader_election.cc:302] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87ff88918faf4016956388144adb6ee0; no voters:
I20260927 22:51:50.438035 6048 leader_election.cc:288] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:50.438176 6053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:50.438822 6053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [term 1 LEADER]: Becoming Leader. State: Replica: 87ff88918faf4016956388144adb6ee0, State: Running, Role: LEADER
I20260927 22:51:50.439440 6053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87ff88918faf4016956388144adb6ee0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35721 } }
I20260927 22:51:50.439978 6048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:50.442577 6055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87ff88918faf4016956388144adb6ee0. Latest consensus state: current_term: 1 leader_uuid: "87ff88918faf4016956388144adb6ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87ff88918faf4016956388144adb6ee0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35721 } } }
I20260927 22:51:50.443001 6055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:50.443485 6054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87ff88918faf4016956388144adb6ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87ff88918faf4016956388144adb6ee0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35721 } } }
I20260927 22:51:50.443763 6054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:50.444626 6061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:50.449553 6061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:50.457211 6061 catalog_manager.cc:1347] Generated new cluster ID: 21dbc63f3101452ba96529a5c7ad6891
I20260927 22:51:50.457329 6061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:50.472325 6061 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:50.474006 6061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:50.493903 6061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87ff88918faf4016956388144adb6ee0: Generated new TSK 0
I20260927 22:51:50.494746 6061 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:51:50.545804 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35721
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:50.854399 6074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:50.854656 6074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:50.854803 6074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:50.861634 6074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:50.861748 6074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:50.861920 6074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:50.871454 6074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:50.872715 6074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:50.873996 6074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:50.886301 6080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:50.886365 6079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:50.886408 6082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:50.887409 6074 server_base.cc:1034] running on GCE node
I20260927 22:51:50.887982 6074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:50.888890 6074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:50.890097 6074 hybrid_clock.cc:648] HybridClock initialized: now 1790549510890066 us; error 55 us; skew 500 ppm
I20260927 22:51:50.890405 6074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:50 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:50.893399 6074 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:51:50.894829 6074 webserver.cc:460] Webserver started at http://127.0.97.1:42797/ using document root <none> and password file <none>
I20260927 22:51:50.895471 6074 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:50.895603 6074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:50.895887 6074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:50.898650 6074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "bfd4592fa3b34ca79019cc2ad2fe5de0"
format_stamp: "Formatted at 2026-09-27 22:51:50 on dist-test-slave-df53"
server_key: "694629f2ef6c5f6ac9f6bee5bba6d839"
server_key_iv: "859fab4c0c921685e6f24f77f1739d4b"
server_key_version: "encryptionkey@0"
I20260927 22:51:50.899384 6074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bfd4592fa3b34ca79019cc2ad2fe5de0"
format_stamp: "Formatted at 2026-09-27 22:51:50 on dist-test-slave-df53"
server_key: "694629f2ef6c5f6ac9f6bee5bba6d839"
server_key_iv: "859fab4c0c921685e6f24f77f1739d4b"
server_key_version: "encryptionkey@0"
I20260927 22:51:50.904171 6074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:50.907461 6089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:50.908772 6074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:50.908933 6074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "bfd4592fa3b34ca79019cc2ad2fe5de0"
format_stamp: "Formatted at 2026-09-27 22:51:50 on dist-test-slave-df53"
server_key: "694629f2ef6c5f6ac9f6bee5bba6d839"
server_key_iv: "859fab4c0c921685e6f24f77f1739d4b"
server_key_version: "encryptionkey@0"
I20260927 22:51:50.909123 6074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:50.942499 6074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:50.946874 6074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:50.947151 6074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:50.948010 6074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:50.949241 6074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:50.949337 6074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:50.949472 6074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:50.949540 6074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:50.980608 6074 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42707
I20260927 22:51:50.980654 6201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42707 every 8 connection(s)
I20260927 22:51:50.981891 6074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:50.984402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 6074
I20260927 22:51:50.984622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:50.985420 388 external_mini_cluster.cc:1428] Setting key 436c03d8c5467540e3dc94cf918cf213
Sep 27 22:51:50 dist-test-slave-df53 krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:51.004338 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:50.984411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32887 (local address 127.0.97.62:35721)
0927 22:51:50.984869 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:50.984877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:50.985929 (+ 1052us) server_negotiation.cc:407] Connection header received
0927 22:51:50.987865 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:50.987869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:50.987926 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:50.988012 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:50.990043 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:50.990750 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:50.993204 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:50.993735 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:50.993833 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:50.997566 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:50.997575 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:50.997583 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:50.997653 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:51.001035 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:51.001773 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:51.001782 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:51.001788 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:51.001943 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:51.002371 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:51.002379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:51.002386 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:51.002777 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:51.002935 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:51.003573 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 22:51:51.003823 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":214,"threads_started":1}
I20260927 22:51:51.004938 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:50.984910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35721 (local address 127.0.97.1:32887)
0927 22:51:50.985759 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:50.985806 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:50.987589 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:50.988276 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:50.988291 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:50.989052 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:50.989889 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:50.989916 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:50.990894 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:50.990902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:50.993049 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:50.993066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:50.993954 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:50.993962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:50.994270 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:50.994953 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:50.994979 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:50.997350 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:51.001187 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:51.001193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:51.001209 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:51.001668 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:51.002074 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:51.002077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:51.002079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:51.002254 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:51.002908 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:51.002913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:51.003206 (+ 293us) client_negotiation.cc:769] Sending connection context
0927 22:51:51.003425 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 22:51:51.003785 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":310,"threads_started":1}
I20260927 22:51:51.006376 6202 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35721
I20260927 22:51:51.006659 6202 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:51:51.007356 6202 heartbeater.cc:511] Master 127.0.97.62:35721 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:51:51.009373 6013 ts_manager.cc:194] Registered new tserver with Master: bfd4592fa3b34ca79019cc2ad2fe5de0 (127.0.97.1:42707)
I20260927 22:51:51.011475 6013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:51:51.033226 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35721
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:51.337711 6208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:51.337982 6208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:51.338127 6208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:51.345202 6208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:51.345335 6208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:51.345481 6208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:51.354604 6208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:51.355757 6208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:51.356984 6208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:51.369518 6214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:51.369625 6213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:51.369625 6216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:51.370827 6208 server_base.cc:1034] running on GCE node
I20260927 22:51:51.371407 6208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:51.372247 6208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:51.373430 6208 hybrid_clock.cc:648] HybridClock initialized: now 1790549511373372 us; error 50 us; skew 500 ppm
I20260927 22:51:51.373755 6208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:51 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:51.376737 6208 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:51:51.378046 6208 webserver.cc:460] Webserver started at http://127.0.97.2:46087/ using document root <none> and password file <none>
I20260927 22:51:51.378693 6208 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:51.378789 6208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:51.379014 6208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:51.381723 6208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "c2cd49220fb64460a908d4e77ae77e6c"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-df53"
server_key: "8ec1c86b796a6fde1dfbc00ae4bff332"
server_key_iv: "3ed0cec031d0e99ddea60242c8de89a7"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.382422 6208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c2cd49220fb64460a908d4e77ae77e6c"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-df53"
server_key: "8ec1c86b796a6fde1dfbc00ae4bff332"
server_key_iv: "3ed0cec031d0e99ddea60242c8de89a7"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.387382 6208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:51:51.390439 6223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.391570 6208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:51:51.391693 6208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "c2cd49220fb64460a908d4e77ae77e6c"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-df53"
server_key: "8ec1c86b796a6fde1dfbc00ae4bff332"
server_key_iv: "3ed0cec031d0e99ddea60242c8de89a7"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.391844 6208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:51.403465 6208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:51.407480 6208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:51.407745 6208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:51.408577 6208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:51.409858 6208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:51.409929 6208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.410076 6208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:51.410125 6208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.442515 6208 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41847
I20260927 22:51:51.442557 6335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41847 every 8 connection(s)
I20260927 22:51:51.443713 6208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:51.449913 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 6208
I20260927 22:51:51.450085 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:51.450598 388 external_mini_cluster.cc:1428] Setting key a4ebe241534045f437d1ea20ce95d918
Sep 27 22:51:51 dist-test-slave-df53 krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:51.465040 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:51.446234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41723 (local address 127.0.97.62:35721)
0927 22:51:51.446434 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:51.446438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:51.447574 (+ 1136us) server_negotiation.cc:407] Connection header received
0927 22:51:51.449270 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:51.449274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:51.449350 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:51.449439 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:51.451426 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.452397 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.454406 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.455148 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.455284 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:51.458448 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:51.458456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:51.458463 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:51.458506 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:51.462041 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:51.462686 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:51.462693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:51.462700 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:51.462847 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:51.463216 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:51.463223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:51.463229 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:51.463571 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:51.463717 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:51.464354 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 22:51:51.464605 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:51:51.465555 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:51.446687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35721 (local address 127.0.97.2:41723)
0927 22:51:51.447406 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:51.447451 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:51.448981 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:51.449736 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:51.449749 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:51.450533 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:51.451249 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:51.451268 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.452527 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.452531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:51.454267 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:51.454283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.455339 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.455343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:51.455520 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:51.456168 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:51.456189 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:51.458244 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:51.462165 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:51.462171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:51.462181 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:51.462563 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:51.462945 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:51.462948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:51.462951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:51.463106 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:51.463690 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:51.463695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:51.463952 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 22:51:51.464184 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 22:51:51.464519 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":319,"threads_started":1}
I20260927 22:51:51.466950 6336 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35721
I20260927 22:51:51.467283 6336 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:51.467927 6336 heartbeater.cc:511] Master 127.0.97.62:35721 requested a full tablet report, sending...
I20260927 22:51:51.469343 6013 ts_manager.cc:194] Registered new tserver with Master: c2cd49220fb64460a908d4e77ae77e6c (127.0.97.2:41847)
I20260927 22:51:51.470530 6013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41723
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:51:51.494901 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35721
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:51.805531 6341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:51.805873 6341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:51.805969 6341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:51.812915 6341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:51.813004 6341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:51.813180 6341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:51.822366 6341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:51.823516 6341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:51.824738 6341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:51.836973 6346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:51.836908 6349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:51.837007 6347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:51.837818 6341 server_base.cc:1034] running on GCE node
I20260927 22:51:51.838316 6341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:51.839068 6341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:51.840274 6341 hybrid_clock.cc:648] HybridClock initialized: now 1790549511840232 us; error 53 us; skew 500 ppm
I20260927 22:51:51.840548 6341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:51 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:51.843489 6341 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:51:51.844816 6341 webserver.cc:460] Webserver started at http://127.0.97.3:33235/ using document root <none> and password file <none>
I20260927 22:51:51.845475 6341 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:51.845623 6341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:51.845906 6341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:51.848467 6341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "ed80667d2110476d9e5f3f97ccb8b221"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-df53"
server_key: "12d6c0aa65b403beee28f62e37fa2d3e"
server_key_iv: "4a2a1a0a0ebdfeb7b91a44bf2ac727d1"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.849237 6341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ed80667d2110476d9e5f3f97ccb8b221"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-df53"
server_key: "12d6c0aa65b403beee28f62e37fa2d3e"
server_key_iv: "4a2a1a0a0ebdfeb7b91a44bf2ac727d1"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.854205 6341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:51:51.857355 6356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.858596 6341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:51.858762 6341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "ed80667d2110476d9e5f3f97ccb8b221"
format_stamp: "Formatted at 2026-09-27 22:51:51 on dist-test-slave-df53"
server_key: "12d6c0aa65b403beee28f62e37fa2d3e"
server_key_iv: "4a2a1a0a0ebdfeb7b91a44bf2ac727d1"
server_key_version: "encryptionkey@0"
I20260927 22:51:51.858942 6341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:51.890122 6341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:51.893932 6341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:51.894204 6341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:51.895197 6341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:51.896546 6341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:51.896642 6341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.896783 6341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:51.896857 6341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:51.928161 6341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36635
I20260927 22:51:51.928208 6468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36635 every 8 connection(s)
I20260927 22:51:51.929383 6341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:51.932016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 6341
I20260927 22:51:51.932189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:51.932616 388 external_mini_cluster.cc:1428] Setting key 38fcea804f9e2994c402dc041dd00714
Sep 27 22:51:51 dist-test-slave-df53 krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:51:51.947857 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:51.931794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57817 (local address 127.0.97.62:35721)
0927 22:51:51.932008 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:51.932015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:51.933300 (+ 1285us) server_negotiation.cc:407] Connection header received
0927 22:51:51.935143 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:51.935146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:51.935231 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:51.935331 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:51.936663 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.937365 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.938842 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.939220 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.939311 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:51.942542 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:51.942547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:51.942550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:51.942578 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:51.944913 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:51.945670 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:51.945673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:51.945676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:51.945832 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:51.946312 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:51.946317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:51.946319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:51.946622 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:51.946756 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:51.947490 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 22:51:51.947633 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:51:51.948683 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:51.932346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35721 (local address 127.0.97.3:57817)
0927 22:51:51.933135 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:51.933188 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:51.934868 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:51.935535 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:51.935543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:51.936000 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:51.936534 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:51.936550 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.937484 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.937488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:51.938725 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:51.938735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.939383 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.939387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:51.939573 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:51.940266 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:51.940288 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:51.942365 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:51.945020 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:51.945026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:51.945067 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:51.945517 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:51.945958 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:51.945963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:51.945967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:51.946170 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:51.946747 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:51.946754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:51.947098 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 22:51:51.947397 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 22:51:51.947716 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":358,"threads_started":1}
I20260927 22:51:51.949853 6469 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35721
I20260927 22:51:51.950109 6469 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:51.950747 6469 heartbeater.cc:511] Master 127.0.97.62:35721 requested a full tablet report, sending...
I20260927 22:51:51.951982 6013 ts_manager.cc:194] Registered new tserver with Master: ed80667d2110476d9e5f3f97ccb8b221 (127.0.97.3:36635)
I20260927 22:51:51.952749 6013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57817
I20260927 22:51:51.959062 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/test-admin.keytab.
Sep 27 22:51:51 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:51.984680 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 22:51:51.989423 6476 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:51:51.987154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.0.97.1:42707)
0927 22:51:51.987730 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:51.987739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:51.987769 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:51:51.987840 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:51.987848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:51.988036 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:51.988210 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:51.989007 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41644: BlockingRecv error: recv got EOF from 127.0.0.1:41644 (error 108)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":292,"threads_started":1}
W20260927 22:51:51.989701 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5cc)
Sep 27 22:51:51 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:51.991109 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:51:51 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:51.992379 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:51:51 dist-test-slave-df53 krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:52.006693 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:51.993804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.0.97.1:42707)
0927 22:51:51.993989 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:51.993996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:51.994259 (+ 263us) server_negotiation.cc:407] Connection header received
0927 22:51:51.994426 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:51.994431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:51.994489 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:51.994584 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:51.995505 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.996495 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.997708 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:51.998229 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:51.998354 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.001666 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.001677 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.001696 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.001730 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.004151 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.004759 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.004767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.004769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.004881 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.005226 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.005233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.005236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.005548 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.005881 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.006198 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.006439 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 22:51:52.011919 6476 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:51:52.010305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.0.97.1:42707)
0927 22:51:52.010491 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.010497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.010692 (+ 195us) server_negotiation.cc:407] Connection header received
0927 22:51:52.010852 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.010857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.010917 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.011008 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.011672 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41664: BlockingRecv error: recv got EOF from 127.0.0.1:41664 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 22:51:52.012076 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:52.013547 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:52.014806 388 init.cc:295] Successfully reacquired a new kerberos TGT
I20260927 22:51:52.015065 6202 heartbeater.cc:503] Master 127.0.97.62:35721 was elected leader, sending a full tablet report...
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:52.027666 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.016123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.0.97.1:42707)
0927 22:51:52.016264 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.016269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.016597 (+ 328us) server_negotiation.cc:407] Connection header received
0927 22:51:52.016739 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.016743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.016786 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.016855 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.017729 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.018382 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.019535 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.019938 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.020027 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.023107 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.023112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.023115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.023140 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.025550 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.026165 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.026168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.026170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.026277 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.026622 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.026625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.026626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.026848 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.027012 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.027227 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.027431 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 22:51:52.031366 6476 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:51:52.029858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.0.97.1:42707)
0927 22:51:52.029975 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.029979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.030357 (+ 378us) server_negotiation.cc:407] Connection header received
0927 22:51:52.030459 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.030462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.030516 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.030588 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.031171 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41688: BlockingRecv error: recv got EOF from 127.0.0.1:41688 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 22:51:52.031505 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549465302610-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:52.032884 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[5968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:52.034139 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:51:52.046590 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.035191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.0.97.1:42707)
0927 22:51:52.035365 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.035371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.035614 (+ 243us) server_negotiation.cc:407] Connection header received
0927 22:51:52.035730 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.035733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.035776 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.035840 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.036654 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.037277 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.038440 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.038804 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.038887 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.042002 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.042006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.042009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.042034 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.044385 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.044934 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.044938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.044940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.045099 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.045440 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.045443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.045445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.045729 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.045944 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.046153 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.046319 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:51:52.047852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 6074
I20260927 22:51:52.058389 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.048393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.0.97.1:42707)
0927 22:51:52.048556 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.048563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.048801 (+ 238us) server_negotiation.cc:407] Connection header received
0927 22:51:52.048960 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.048969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.049013 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.049104 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.049987 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.050597 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.051747 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.052106 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.052230 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.054108 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.054112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.054119 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.054144 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.056130 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.056717 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.056723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.056726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.056839 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.057186 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.057192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.057196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.057408 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.057670 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.057936 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.058110 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 22:51:52.058826 6197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:52.138926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 6074
I20260927 22:51:52.154567 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 6208
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:51:52.168911 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.155391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.0.97.2:41847)
0927 22:51:52.155875 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.155882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.155907 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:51:52.155961 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.155967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.156169 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.156330 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.157266 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.158311 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.159488 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.160019 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.160161 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.163932 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.163941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.163954 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.163984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.166700 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.167264 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.167270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.167272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.167378 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.167699 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.167702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.167703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.167991 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.168200 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.168432 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.168598 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":236,"threads_started":1}
I20260927 22:51:52.170264 6331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:52.250142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 6208
I20260927 22:51:52.265265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 6341
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[5968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:51:52.279603 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:52.266240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.0.97.3:36635)
0927 22:51:52.266784 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:52.266791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:52.266817 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:51:52.266870 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:52.266876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:52.267101 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:52.267224 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:52.268169 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.269175 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.270414 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:52.270974 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:52.271072 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:52.274708 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:52.274718 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:52.274730 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:52.274759 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:52.277263 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.277903 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.277910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.277913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.278061 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:52.278379 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:52.278383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:52.278385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:52.278701 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:52.278925 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:52.279155 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 22:51:52.279341 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":279,"threads_started":1}
I20260927 22:51:52.280777 6464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:52.378736 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 6341
I20260927 22:51:52.393978 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 5981
I20260927 22:51:52.395072 6043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:52.495884 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 5981
2026-09-27T22:51:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3754 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[6496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[6496](info): set up 2 sockets
Sep 27 22:51:52 dist-test-slave-df53 krb5kdc[6496](info): commencing operation
krb5kdc: starting...
W20260927 22:51:53.583021 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:51:53 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:51:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 22:51:53.772501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35383
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:54.076480 6509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:54.076740 6509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:54.076828 6509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:54.083866 6509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:54.083981 6509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:54.084028 6509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:54.084091 6509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:54.084136 6509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:54.093524 6509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:54.094731 6509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:54.095880 6509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:54.107035 6514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:54.107199 6517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:54.107057 6515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:54.108223 6509 server_base.cc:1034] running on GCE node
I20260927 22:51:54.108968 6509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:54.110303 6509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:54.111500 6509 hybrid_clock.cc:648] HybridClock initialized: now 1790549514111443 us; error 53 us; skew 500 ppm
I20260927 22:51:54.111830 6509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:54 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549514, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:54.114854 6509 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 22:51:54.116232 6509 webserver.cc:460] Webserver started at http://127.0.97.62:41551/ using document root <none> and password file <none>
I20260927 22:51:54.116863 6509 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:54.117004 6509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:54.117292 6509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:54.120105 6509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "6be24da53c1245a8afab32e4c229cc0f"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-df53"
server_key: "8d48cc96d5f5997710690c74933b5454"
server_key_iv: "d5e0c6f2c0fc4dc893ee4ac73ada2209"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.120832 6509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "6be24da53c1245a8afab32e4c229cc0f"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-df53"
server_key: "8d48cc96d5f5997710690c74933b5454"
server_key_iv: "d5e0c6f2c0fc4dc893ee4ac73ada2209"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.125680 6509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 22:51:54.128847 6524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.130182 6509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:51:54.130308 6509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "6be24da53c1245a8afab32e4c229cc0f"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-df53"
server_key: "8d48cc96d5f5997710690c74933b5454"
server_key_iv: "d5e0c6f2c0fc4dc893ee4ac73ada2209"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.130460 6509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:54.146697 6509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:54.150369 6509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:54.150605 6509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:54.167937 6509 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35383
I20260927 22:51:54.167999 6575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35383 every 8 connection(s)
I20260927 22:51:54.169224 6509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:51:54.172690 6576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:54.179136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 6509
I20260927 22:51:54.179333 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:51:54.179956 388 external_mini_cluster.cc:1428] Setting key a762e6bcffdfb35d3a43265eb9117e7e
I20260927 22:51:54.182843 6576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f: Bootstrap starting.
I20260927 22:51:54.186729 6576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:54.187829 6576 log.cc:824] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:54.190662 6576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f: No bootstrap required, opened a new log
Sep 27 22:51:54 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:51:54.200057 6576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } }
I20260927 22:51:54.200590 6576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:54.200721 6576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be24da53c1245a8afab32e4c229cc0f, State: Initialized, Role: FOLLOWER
I20260927 22:51:54.201449 6576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } }
I20260927 22:51:54.201774 6576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:51:54.201924 6576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:51:54.202114 6576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:54.203924 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:54.181908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.0.97.62:35383)
0927 22:51:54.182717 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:54.182731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:54.182778 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:51:54.185194 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:54.185220 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:54.185697 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:54.186121 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:54.187494 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.189261 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.190530 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.191561 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.191731 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:54.195410 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:54.195420 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:54.195438 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:54.195492 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:54.199376 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:54.200053 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:54.200061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:54.200070 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:54.200258 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:54.200718 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:54.200724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:54.200728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:54.201401 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:54.201736 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:54.202615 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 22:51:54.202962 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":333,"threads_started":1}
I20260927 22:51:54.204504 6576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } }
I20260927 22:51:54.204995 6576 leader_election.cc:302] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6be24da53c1245a8afab32e4c229cc0f; no voters:
I20260927 22:51:54.205755 6576 leader_election.cc:288] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:51:54.205842 6581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:51:54.206756 6581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 1 LEADER]: Becoming Leader. State: Replica: 6be24da53c1245a8afab32e4c229cc0f, State: Running, Role: LEADER
I20260927 22:51:54.207345 6581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } }
I20260927 22:51:54.208014 6576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:54.210207 6583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6be24da53c1245a8afab32e4c229cc0f. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } }
I20260927 22:51:54.210996 6583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:54.210707 6582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } }
I20260927 22:51:54.211472 6582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:54.214668 6588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:51:54.220160 6588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:51:54.228593 6588 catalog_manager.cc:1347] Generated new cluster ID: a5c4f88810be476fa78002bf583faa9c
I20260927 22:51:54.228714 6588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:51:54.249051 6588 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:51:54.250814 6588 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:51:54.264281 6588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f: Generated new TSK 0
I20260927 22:51:54.265204 6588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 22:51:54.329089 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:54.659670 6602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:54.659909 6602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:54.660059 6602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:54.666939 6602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:54.667039 6602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:54.667236 6602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:54.676409 6602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:54.677588 6602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:54.678893 6602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:54.691380 6607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:54.691406 6608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:54.691596 6610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:54.692214 6602 server_base.cc:1034] running on GCE node
I20260927 22:51:54.692776 6602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:54.693660 6602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:54.694885 6602 hybrid_clock.cc:648] HybridClock initialized: now 1790549514694826 us; error 55 us; skew 500 ppm
I20260927 22:51:54.695186 6602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:54 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549514, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:54.698230 6602 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:51:54.699465 6602 webserver.cc:460] Webserver started at http://127.0.97.1:33035/ using document root <none> and password file <none>
I20260927 22:51:54.700050 6602 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:54.700152 6602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:54.700387 6602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:54.703471 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "820c3e17809a41d2847c3da4760d7aa0"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-df53"
server_key: "ecc38c6bb0ba55f5a0a7265de509c6d3"
server_key_iv: "f0e354985f3362be3ecb26e62cc145ff"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.704180 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "820c3e17809a41d2847c3da4760d7aa0"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-df53"
server_key: "ecc38c6bb0ba55f5a0a7265de509c6d3"
server_key_iv: "f0e354985f3362be3ecb26e62cc145ff"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.709156 6602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:51:54.712510 6617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.713819 6602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:51:54.713965 6602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "820c3e17809a41d2847c3da4760d7aa0"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-df53"
server_key: "ecc38c6bb0ba55f5a0a7265de509c6d3"
server_key_iv: "f0e354985f3362be3ecb26e62cc145ff"
server_key_version: "encryptionkey@0"
I20260927 22:51:54.714155 6602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:54.747542 6602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:54.751330 6602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:54.751610 6602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:54.752508 6602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:54.753785 6602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:54.753856 6602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.754002 6602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:54.754091 6602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:54.785777 6602 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44033
I20260927 22:51:54.785846 6729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44033 every 8 connection(s)
I20260927 22:51:54.786996 6602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:54.789649 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 6602
I20260927 22:51:54.789848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:51:54.790701 388 external_mini_cluster.cc:1428] Setting key c6e9a6419a907fdf8a8d0c77cf23ecf9
Sep 27 22:51:54 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549514, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:51:54.806262 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:54.789934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58303 (local address 127.0.97.62:35383)
0927 22:51:54.790381 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:54.790385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:54.791176 (+ 791us) server_negotiation.cc:407] Connection header received
0927 22:51:54.793119 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:54.793125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:54.793215 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:54.793328 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:54.794837 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.795546 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.797082 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.797450 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.797583 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:54.800793 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:54.800798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:54.800801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:54.800830 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:54.803400 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:54.804098 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:54.804102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:54.804104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:54.804211 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:54.804614 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:54.804619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:54.804622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:54.804864 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:54.805012 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:54.805685 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 22:51:54.805899 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":236,"threads_started":1}
I20260927 22:51:54.807149 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:54.790205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.1:58303)
0927 22:51:54.791017 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:54.791060 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:54.792868 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:54.793558 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:54.793566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:54.794126 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:54.794674 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:54.794691 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.795709 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.795715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:54.796915 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:54.796926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:54.797694 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:54.797701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:54.797919 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:54.798575 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:54.798601 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:54.800570 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:54.803533 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:54.803541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:54.803564 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:54.803946 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:54.804315 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:54.804322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:54.804325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:54.804496 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:54.804986 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:54.804994 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:54.805298 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 22:51:54.805562 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 22:51:54.806079 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":339,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:51:54.808756 6730 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35383
I20260927 22:51:54.809044 6730 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:54.809816 6730 heartbeater.cc:511] Master 127.0.97.62:35383 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260927 22:51:54.811914 6541 ts_manager.cc:194] Registered new tserver with Master: 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033)
I20260927 22:51:54.813298 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 22:51:54.834389 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:55.139174 6736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:55.139464 6736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:55.139638 6736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:55.146481 6736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:55.146615 6736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:55.146795 6736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:55.155836 6736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:55.156993 6736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:55.158245 6736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:55.171741 6744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:55.171707 6742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:55.172492 6741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:55.175168 6736 server_base.cc:1034] running on GCE node
I20260927 22:51:55.175733 6736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:55.176669 6736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:55.177892 6736 hybrid_clock.cc:648] HybridClock initialized: now 1790549515177849 us; error 36 us; skew 500 ppm
I20260927 22:51:55.178223 6736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:55 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549515, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:55.181389 6736 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:51:55.186708 6736 webserver.cc:460] Webserver started at http://127.0.97.2:36919/ using document root <none> and password file <none>
I20260927 22:51:55.187389 6736 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:55.187533 6736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:55.187816 6736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:55.190575 6736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "400068953a8d411086ce260244bbdf2d"
format_stamp: "Formatted at 2026-09-27 22:51:55 on dist-test-slave-df53"
server_key: "164dac1355f3f6c6858b3a4970917b73"
server_key_iv: "6165a05d73972ebee7e09a736082b2ef"
server_key_version: "encryptionkey@0"
I20260927 22:51:55.191288 6736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "400068953a8d411086ce260244bbdf2d"
format_stamp: "Formatted at 2026-09-27 22:51:55 on dist-test-slave-df53"
server_key: "164dac1355f3f6c6858b3a4970917b73"
server_key_iv: "6165a05d73972ebee7e09a736082b2ef"
server_key_version: "encryptionkey@0"
I20260927 22:51:55.196127 6736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:55.199321 6751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:55.200552 6736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:51:55.200721 6736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "400068953a8d411086ce260244bbdf2d"
format_stamp: "Formatted at 2026-09-27 22:51:55 on dist-test-slave-df53"
server_key: "164dac1355f3f6c6858b3a4970917b73"
server_key_iv: "6165a05d73972ebee7e09a736082b2ef"
server_key_version: "encryptionkey@0"
I20260927 22:51:55.200914 6736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:55.228430 6736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:55.232053 6736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:55.232340 6736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:55.233234 6736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:55.234619 6736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:55.234723 6736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:55.234823 6736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:55.234865 6736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:55.266661 6736 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35461
I20260927 22:51:55.266742 6863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35461 every 8 connection(s)
I20260927 22:51:55.267884 6736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:55.270416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 6736
I20260927 22:51:55.270597 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:51:55.271003 388 external_mini_cluster.cc:1428] Setting key 3c6786397fd9dcecafa110635abb5159
Sep 27 22:51:55 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549515, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:51:55.288899 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.270269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59921 (local address 127.0.97.62:35383)
0927 22:51:55.270463 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.270470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.271761 (+ 1291us) server_negotiation.cc:407] Connection header received
0927 22:51:55.273533 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.273540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.273644 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.273752 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:55.275291 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.276391 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.277915 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.278669 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.278767 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.282884 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:55.282889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:55.282892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:55.282921 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:55.285518 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:55.286465 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:55.286470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:55.286474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:55.286603 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:55.287125 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:55.287130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:55.287135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:55.287407 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:55.287536 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.288413 (+ 877us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.288657 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:51:55.290349 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.270823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:59921)
0927 22:51:55.271608 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.271652 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:55.273282 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:55.273942 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:55.273950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:55.274429 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:55.275084 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.275110 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.276544 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.276548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.277788 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.277799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.278893 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.278904 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.279289 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.280039 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:55.280062 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:55.282674 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:55.285682 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:55.285694 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:55.285712 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:55.286336 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:55.286720 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:55.286728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:55.286735 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:55.286977 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:55.287534 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:55.287544 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:55.287994 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 22:51:55.288318 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 22:51:55.288911 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":347,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:51:55.291817 6864 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35383
I20260927 22:51:55.292119 6864 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:55.292861 6864 heartbeater.cc:511] Master 127.0.97.62:35383 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260927 22:51:55.294399 6541 ts_manager.cc:194] Registered new tserver with Master: 400068953a8d411086ce260244bbdf2d (127.0.97.2:35461)
I20260927 22:51:55.295166 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 22:51:55.317164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:55.631114 6869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:55.631371 6869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:55.631459 6869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:55.638329 6869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:55.638417 6869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:55.638520 6869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:55.647904 6869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:55.648998 6869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:55.650313 6869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:55.662770 6875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:55.662961 6877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:55.663180 6874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:55.663834 6869 server_base.cc:1034] running on GCE node
I20260927 22:51:55.664404 6869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:55.665277 6869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:55.666541 6869 hybrid_clock.cc:648] HybridClock initialized: now 1790549515666481 us; error 64 us; skew 500 ppm
I20260927 22:51:55.666865 6869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:55 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549515, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:55.676726 6869 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:51:55.678176 6869 webserver.cc:460] Webserver started at http://127.0.97.3:32821/ using document root <none> and password file <none>
I20260927 22:51:55.678831 6869 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:55.678972 6869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:55.679256 6869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:55.682215 6869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "53f009fcb18843b9bebd0475077d28f6"
format_stamp: "Formatted at 2026-09-27 22:51:55 on dist-test-slave-df53"
server_key: "b296102a58073335e322a6be619fcbac"
server_key_iv: "6a95d9520638522ad5947519f35d1e6e"
server_key_version: "encryptionkey@0"
I20260927 22:51:55.682973 6869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "53f009fcb18843b9bebd0475077d28f6"
format_stamp: "Formatted at 2026-09-27 22:51:55 on dist-test-slave-df53"
server_key: "b296102a58073335e322a6be619fcbac"
server_key_iv: "6a95d9520638522ad5947519f35d1e6e"
server_key_version: "encryptionkey@0"
I20260927 22:51:55.687894 6869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:51:55.691293 6884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:55.692517 6869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:51:55.692691 6869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "53f009fcb18843b9bebd0475077d28f6"
format_stamp: "Formatted at 2026-09-27 22:51:55 on dist-test-slave-df53"
server_key: "b296102a58073335e322a6be619fcbac"
server_key_iv: "6a95d9520638522ad5947519f35d1e6e"
server_key_version: "encryptionkey@0"
I20260927 22:51:55.692883 6869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:55.711133 6869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:55.716519 6869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:55.716940 6869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:55.718253 6869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:55.719887 6869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:55.719957 6869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:55.720088 6869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:55.720155 6869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:55.752126 6869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35043
I20260927 22:51:55.752229 6996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35043 every 8 connection(s)
I20260927 22:51:55.753358 6869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:55.754002 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 6869
I20260927 22:51:55.754195 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:51:55.754753 388 external_mini_cluster.cc:1428] Setting key 98bc3a00722d191fc9088c944bb5e186
Sep 27 22:51:55 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549515, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:51:55.772670 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.756188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38487 (local address 127.0.97.62:35383)
0927 22:51:55.756368 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:55.756373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:55.757483 (+ 1110us) server_negotiation.cc:407] Connection header received
0927 22:51:55.759432 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:55.759437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:55.759534 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:55.759639 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:55.761170 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.762024 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.763515 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.763904 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.764069 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.767305 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:55.767310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:55.767314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:55.767352 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:55.769951 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:55.770760 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:55.770763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:55.770765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:55.770883 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:55.771237 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:55.771240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:55.771242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:55.771486 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:55.771589 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:55.772259 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 22:51:55.772409 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:51:55.773371 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:55.756596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.3:38487)
0927 22:51:55.757347 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:55.757383 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:55.759183 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:55.759871 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:55.759879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:55.760387 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:55.760942 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.760958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.762193 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.762198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.763373 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:55.763384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:55.764175 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:55.764183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:55.764393 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:55.765020 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:55.765045 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:55.767123 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:55.770081 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:55.770090 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:55.770107 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:55.770654 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:55.770990 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:55.770993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:55.770996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:55.771151 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:55.771584 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:55.771589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:55.771848 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 22:51:55.772070 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 22:51:55.772390 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":427,"threads_started":1}
I20260927 22:51:55.774597 6997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35383
I20260927 22:51:55.774878 6997 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:55.775543 6997 heartbeater.cc:511] Master 127.0.97.62:35383 requested a full tablet report, sending...
I20260927 22:51:55.776841 6541 ts_manager.cc:194] Registered new tserver with Master: 53f009fcb18843b9bebd0475077d28f6 (127.0.97.3:35043)
I20260927 22:51:55.777540 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38487
I20260927 22:51:55.782349 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
I20260927 22:51:55.816262 6730 heartbeater.cc:503] Master 127.0.97.62:35383 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260927 22:51:55.825059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38543
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35383,127.0.97.61:38543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:56.130692 7002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:56.130950 7002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:56.131036 7002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:56.138067 7002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:56.138156 7002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:56.138198 7002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:56.138252 7002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:56.138286 7002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:56.147331 7002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35383,127.0.97.61:38543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
W20260927 22:51:56.147598 7002 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:35383,127.0.97.61:38543'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 22:51:56.148489 7002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:56.149667 7002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:56.160997 7008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:56.161087 7007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:56.161087 7010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:56.161856 7002 server_base.cc:1034] running on GCE node
I20260927 22:51:56.162477 7002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:56.163630 7002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:56.164831 7002 hybrid_clock.cc:648] HybridClock initialized: now 1790549516164781 us; error 31 us; skew 500 ppm
I20260927 22:51:56.165138 7002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:56.168108 7002 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 22:51:56.174631 7002 webserver.cc:460] Webserver started at http://127.0.97.61:41411/ using document root <none> and password file <none>
I20260927 22:51:56.175276 7002 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:56.175398 7002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:56.175691 7002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:56.178421 7002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data/instance:
uuid: "2a73ae77b7f54c929293c60d37f40189"
format_stamp: "Formatted at 2026-09-27 22:51:56 on dist-test-slave-df53"
server_key: "0b41cb15577bdda7ea2a763b8d524653"
server_key_iv: "7754fcadea1043514fef86975dd92508"
server_key_version: "encryptionkey@0"
I20260927 22:51:56.179157 7002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal/instance:
uuid: "2a73ae77b7f54c929293c60d37f40189"
format_stamp: "Formatted at 2026-09-27 22:51:56 on dist-test-slave-df53"
server_key: "0b41cb15577bdda7ea2a763b8d524653"
server_key_iv: "7754fcadea1043514fef86975dd92508"
server_key_version: "encryptionkey@0"
I20260927 22:51:56.183948 7002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:56.187139 7017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:56.188294 7002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:51:56.188469 7002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal
uuid: "2a73ae77b7f54c929293c60d37f40189"
format_stamp: "Formatted at 2026-09-27 22:51:56 on dist-test-slave-df53"
server_key: "0b41cb15577bdda7ea2a763b8d524653"
server_key_iv: "7754fcadea1043514fef86975dd92508"
server_key_version: "encryptionkey@0"
I20260927 22:51:56.188665 7002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:56.212066 7002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:56.215909 7002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:56.216161 7002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:56.233254 7002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38543
I20260927 22:51:56.233271 7068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38543 every 8 connection(s)
I20260927 22:51:56.234618 7002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:51:56.238003 7069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:56.241568 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 7002
I20260927 22:51:56.244977 7069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:51:56.263880 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.248364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.0.97.62:35383)
0927 22:51:56.248567 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.248576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.249307 (+ 731us) server_negotiation.cc:407] Connection header received
0927 22:51:56.249467 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.249472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.249541 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.249652 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.250697 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.251633 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.253406 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.254137 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.254283 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.257869 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.257876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.257887 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.257918 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.261190 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.261928 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.261934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.261938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.262058 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.262412 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.262418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.262422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.262705 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.262847 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.263318 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.263517 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:51:56.264636 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.248384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45250)
0927 22:51:56.249165 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.249201 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:56.249345 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:56.249723 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:56.249727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:56.250327 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:56.250548 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.250556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.251749 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.251756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.253269 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.253280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.254326 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.254331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.254442 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.255004 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:56.255023 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:56.257687 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:56.261309 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.261321 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.261331 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.261813 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.262180 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.262185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.262189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.262309 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.262848 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:56.262856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:56.263022 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 22:51:56.263231 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 22:51:56.263695 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":148,"thread_start_us":375,"threads_started":1}
I20260927 22:51:56.264657 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.243619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.0.97.61:38543)
0927 22:51:56.244331 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.244341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.244378 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:51:56.245894 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.245914 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.246271 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.246562 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.248278 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.249912 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.251385 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.252239 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.252400 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.256814 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.256828 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.256843 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.256881 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.260685 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.261348 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.261358 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.261366 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.261531 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.261944 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.261951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.261954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.262430 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.262714 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.263107 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.263554 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":528,"thread_start_us":345,"threads_started":1}
I20260927 22:51:56.266494 7069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:51:56.279038 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.267120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.0.1:52872)
0927 22:51:56.267273 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.267291 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:56.267387 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:56.267965 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:56.267970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:56.268349 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:56.268532 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.268542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.269449 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.269461 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.270824 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.270837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.271583 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.271589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.271788 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.272291 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:56.272307 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:56.274285 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:56.277120 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.277124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.277126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.277438 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.277860 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.277863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.277865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.277981 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.278408 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:56.278411 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:56.278537 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 22:51:56.278641 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:51:56.278819 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 22:51:56.279233 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.267597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.0.97.61:38543)
0927 22:51:56.267738 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.267742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.267756 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:51:56.267789 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.267792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.267836 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.267907 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.268655 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.269326 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.270977 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.271425 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.271541 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.274434 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.274439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.274442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.274467 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.277009 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.277573 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.277577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.277579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.277763 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.278080 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.278083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.278085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.278315 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.278416 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.278716 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.278955 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":46}
I20260927 22:51:56.283288 7069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: Bootstrap starting.
I20260927 22:51:56.286051 7069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:56.286926 7069 log.cc:824] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:56.289242 7069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: No bootstrap required, opened a new log
I20260927 22:51:56.296002 7069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } }
I20260927 22:51:56.296322 7069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:56.296396 7069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a73ae77b7f54c929293c60d37f40189, State: Initialized, Role: FOLLOWER
I20260927 22:51:56.296839 7069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } }
I20260927 22:51:56.297559 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } } }
I20260927 22:51:56.297873 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:51:56.298223 6864 heartbeater.cc:503] Master 127.0.97.62:35383 was elected leader, sending a full tablet report...
I20260927 22:51:56.298617 7069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:51:56.307322 7086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:51:56.307452 7086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:51:56.309253 7002 master_runner.cc:428] Detected that this master 2a73ae77b7f54c929293c60d37f40189 is joining an existing cluster
I20260927 22:51:56.309415 7002 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:38543
I20260927 22:51:56.334630 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.313470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.0.97.62:35383)
0927 22:51:56.313671 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.313676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.318226 (+ 4550us) server_negotiation.cc:407] Connection header received
0927 22:51:56.318473 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.318481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.318565 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.318663 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.319985 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.321191 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.323746 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.324415 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.324543 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.327199 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.327206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.327211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.327247 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.330914 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.332018 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.332026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.332032 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.332190 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.332658 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.332665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.332671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.333043 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.333177 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.333935 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.334167 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:51:56.336098 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.313030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45258)
0927 22:51:56.318097 (+ 5067us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.318117 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:56.318344 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:56.318681 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:56.318685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:56.319379 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:56.319806 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.319820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.321773 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.321780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.323585 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.323602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.324789 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.324796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.325092 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.325778 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:56.325802 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:56.327014 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:56.331064 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.331072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.331078 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.331881 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.332329 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.332336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.332342 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.332538 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.333260 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:56.333268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:56.333493 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 22:51:56.335038 (+ 1545us) client_negotiation.cc:240] Negotiation successful
0927 22:51:56.335455 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4933,"mutex_wait_us":63,"thread_start_us":4784,"threads_started":1}
I20260927 22:51:56.338709 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.313256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.0.1:52874)
0927 22:51:56.321140 (+ 7884us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.321158 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:56.321279 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:56.323767 (+ 2488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:56.323775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:56.324449 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:56.324797 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.324812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.326546 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.326553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.328126 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.328142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.329621 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.329628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.329831 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.330457 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:56.330478 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:56.331541 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:56.335980 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.335998 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.336002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.336530 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.336897 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.336906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.336910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.337060 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.337709 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:56.337714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:56.337847 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 22:51:56.337965 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:51:56.338268 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7771,"mutex_wait_us":1,"thread_start_us":8087,"threads_started":1}
I20260927 22:51:56.339607 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.313303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.0.97.61:38543)
0927 22:51:56.313492 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.313499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.321308 (+ 7809us) server_negotiation.cc:407] Connection header received
0927 22:51:56.321372 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.321379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.321454 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.321548 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.324954 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.326325 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.328273 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.328981 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.329376 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.331728 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.331736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.331742 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.331777 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.335849 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.336652 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.336657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.336661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.336785 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.337186 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.337191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.337196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.337542 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.338905 (+ 1363us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.339052 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.339248 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":62}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549514, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:51:56.359638 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.343507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.0.97.61:38543)
0927 22:51:56.344324 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.344331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.344352 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:51:56.344438 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.344446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.344564 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.344678 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.345969 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.346706 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.348663 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.349448 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.349624 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.353050 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.353055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.353058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.353084 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.356481 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.357316 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.357326 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.357330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.357470 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.357941 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.357946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.357949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.358302 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.358442 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.359013 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.359277 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687}
I20260927 22:51:56.359778 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.343217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.0.1:52878)
0927 22:51:56.343855 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.343893 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:56.344076 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:56.344751 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:56.344759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:56.345555 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:56.345832 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.345843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.346842 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.346848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.348476 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.348491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.349641 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.349649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.349848 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.350495 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:56.350518 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:56.352870 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:56.356617 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.356630 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.356640 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.357196 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.357629 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.357637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.357643 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.357822 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.358446 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:56.358456 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:56.358685 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 22:51:56.358884 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 22:51:56.359219 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":354,"threads_started":1}
I20260927 22:51:56.361903 6541 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:38543: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2a73ae77b7f54c929293c60d37f40189"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 38543
}
attrs {
promote: true
}
}
dest_uuid: "6be24da53c1245a8afab32e4c229cc0f"
cas_config_opid_index: -1
I20260927 22:51:56.363123 6541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } }
I20260927 22:51:56.367796 7044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:56.367266 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2a73ae77b7f54c929293c60d37f40189 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } } }
I20260927 22:51:56.371182 7044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Refusing update from remote peer 6be24da53c1245a8afab32e4c229cc0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 22:51:56.372048 7099 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260927 22:51:56.374938 7098 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:38543
I20260927 22:51:56.374998 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46589
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35383,127.0.97.61:38543,127.0.97.60:46589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:51:56.375564 7002 master.cc:537] Master@127.0.97.61:38543 shutting down...
I20260927 22:51:56.375550 7099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } } }
I20260927 22:51:56.376585 7099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: This master's current role is: LEADER
W20260927 22:51:56.383000 6527 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 22:51:56.387172 7002 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:56.387560 7002 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:56.387738 7002 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: stopping tablet replica
I20260927 22:51:56.395022 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.394528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.0.1:52882)
0927 22:51:56.394710 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.394837 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:51:56.395278 6527 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f -> Peer 2a73ae77b7f54c929293c60d37f40189 (127.0.97.61:38543): Couldn't send request to peer 2a73ae77b7f54c929293c60d37f40189. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:51:56.404135 7002 master.cc:552] Master@127.0.97.61:38543 shutdown complete.
I20260927 22:51:56.404223 7002 master_runner.cc:305] Clearing existing system tablet
I20260927 22:51:56.404655 7002 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:56.404745 7002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:56.407823 7002 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:51:56.409572 7104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:56.409900 7002 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:56.410230 7002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 22:51:56.410310 7002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal
uuid: "2a73ae77b7f54c929293c60d37f40189"
format_stamp: "Formatted at 2026-09-27 22:51:56 on dist-test-slave-df53"
server_key: "0b41cb15577bdda7ea2a763b8d524653"
server_key_iv: "7754fcadea1043514fef86975dd92508"
server_key_version: "encryptionkey@0"
I20260927 22:51:56.415226 7002 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:56.421880 7002 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:51:56.422045 7002 log.cc:1196] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 22:51:56.422484 7002 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: Deleting consensus metadata
I20260927 22:51:56.422752 7002 master_runner.cc:315] Copying system tablet from 127.0.97.62:35383
I20260927 22:51:56.425094 7002 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:35383
I20260927 22:51:56.444813 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.425907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45266)
0927 22:51:56.429977 (+ 4070us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.430001 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:56.430209 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:56.430638 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:56.430643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:56.431154 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:56.431415 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.431427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.432880 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.432886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.434392 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.434405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.435404 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.435416 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.435583 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.436302 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:56.436325 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:56.437342 (+ 1017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:56.441432 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.441439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.441444 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.442063 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.442572 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.442578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.442583 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.442776 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.443618 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:56.443624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:56.443833 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 22:51:56.444045 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 22:51:56.444381 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3937,"thread_start_us":3824,"threads_started":1}
I20260927 22:51:56.445742 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.425891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.0.97.62:35383)
0927 22:51:56.426091 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.426097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.430099 (+ 4002us) server_negotiation.cc:407] Connection header received
0927 22:51:56.430385 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.430391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.430487 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.430601 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.431568 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.432733 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.434538 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.435246 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.435392 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.437574 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.437580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.437584 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.437645 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.441243 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.442230 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.442238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.442242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.442397 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.442923 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.442931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.442935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.443357 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.443525 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.445187 (+ 1662us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.445399 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:51:56.446516 6561 tablet_copy_service.cc:140] P 6be24da53c1245a8afab32e4c229cc0f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2a73ae77b7f54c929293c60d37f40189 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:45266)
I20260927 22:51:56.446748 6561 tablet_copy_service.cc:161] P 6be24da53c1245a8afab32e4c229cc0f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2a73ae77b7f54c929293c60d37f40189 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:45266: session id = 2a73ae77b7f54c929293c60d37f40189-00000000000000000000000000000000
I20260927 22:51:56.450795 6561 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:51:56.452761 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:51:56.459688 7002 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: tablet copy: Starting download of 0 data blocks...
I20260927 22:51:56.459991 7002 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: tablet copy: Starting download of 1 WAL segments...
I20260927 22:51:56.462565 7002 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:51:56.466642 6561 tablet_copy_service.cc:342] P 6be24da53c1245a8afab32e4c229cc0f: Request end of tablet copy session 2a73ae77b7f54c929293c60d37f40189-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:45266
I20260927 22:51:56.466826 6561 tablet_copy_service.cc:433] P 6be24da53c1245a8afab32e4c229cc0f: ending tablet copy session 2a73ae77b7f54c929293c60d37f40189-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2a73ae77b7f54c929293c60d37f40189
I20260927 22:51:56.479254 7002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:56.481834 7111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:56.482682 7112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:56.490175 7114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:56.494894 7002 server_base.cc:1034] running on GCE node
I20260927 22:51:56.495241 7002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:56.497803 7002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:56.499034 7002 hybrid_clock.cc:648] HybridClock initialized: now 1790549516498961 us; error 41 us; skew 500 ppm
I20260927 22:51:56.499244 7002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:56.501441 7002 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 22:51:56.505218 7002 webserver.cc:460] Webserver started at http://127.0.97.61:37685/ using document root <none> and password file <none>
I20260927 22:51:56.505716 7002 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:56.505795 7002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:56.508669 7002 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 22:51:56.510488 7121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:56.510969 7002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 22:51:56.511193 7002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal
uuid: "2a73ae77b7f54c929293c60d37f40189"
format_stamp: "Formatted at 2026-09-27 22:51:56 on dist-test-slave-df53"
server_key: "0b41cb15577bdda7ea2a763b8d524653"
server_key_iv: "7754fcadea1043514fef86975dd92508"
server_key_version: "encryptionkey@0"
I20260927 22:51:56.511420 7002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:56.520334 7002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:56.524276 7002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:56.544847 7002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38543
I20260927 22:51:56.544895 7172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38543 every 8 connection(s)
I20260927 22:51:56.546151 7002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:51:56.551718 7173 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:51:56.555740 7173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: Bootstrap starting.
I20260927 22:51:56.571121 7173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:51:56.571584 7173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: Bootstrap complete.
I20260927 22:51:56.572605 7173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } }
I20260927 22:51:56.572914 7173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a73ae77b7f54c929293c60d37f40189, State: Initialized, Role: LEARNER
I20260927 22:51:56.573148 7173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } }
I20260927 22:51:56.574194 7173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:56.574020 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } } }
I20260927 22:51:56.574523 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:51:56.583143 7187 catalog_manager.cc:1259] Loaded cluster ID: a5c4f88810be476fa78002bf583faa9c
I20260927 22:51:56.583253 7187 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: loading cluster ID for follower catalog manager: success
I20260927 22:51:56.585520 7187 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: acquiring CA information for follower catalog manager: success
I20260927 22:51:56.587270 7187 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:51:56.763447 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.757146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.0.1:52884)
0927 22:51:56.757409 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.757430 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:56.757579 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:56.758412 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:56.758418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:56.758447 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:56.758635 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.758644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.760113 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.760121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.761341 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.761361 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.762607 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.762613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.762825 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.762892 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 22:51:56.762999 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:51:56.763141 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 22:51:56.763518 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.757276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.0.97.61:38543)
0927 22:51:56.758111 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.758115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.758134 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:51:56.758188 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.758191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.758249 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.758376 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:56.758786 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.759960 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.761507 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.762432 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.762590 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.762838 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.763022 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.763159 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":574,"threads_started":1}
I20260927 22:51:56.764595 7148 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 22:51:56.765125 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6be24da53c1245a8afab32e4c229cc0f. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } } }
I20260927 22:51:56.765383 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:51:56.779809 6997 heartbeater.cc:503] Master 127.0.97.62:35383 was elected leader, sending a full tablet report...
W20260927 22:51:56.801293 7101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:56.801558 7101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:56.801717 7101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:56.809360 7101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:51:56.809454 7101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:56.809540 7101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:56.809649 7101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:51:56.809697 7101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:51:56.820250 7101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35383,127.0.97.61:38543,127.0.97.60:46589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:56.821483 7101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:56.822767 7101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:56.833930 7193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:56.833901 7194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:56.833844 7196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:56.834798 7101 server_base.cc:1034] running on GCE node
I20260927 22:51:56.835846 7101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:56.837029 7101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:56.838215 7101 hybrid_clock.cc:648] HybridClock initialized: now 1790549516838174 us; error 33 us; skew 500 ppm
I20260927 22:51:56.838541 7101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:56.841472 7101 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 22:51:56.842756 7101 webserver.cc:460] Webserver started at http://127.0.97.60:33323/ using document root <none> and password file <none>
I20260927 22:51:56.843381 7101 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:56.843516 7101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:56.843796 7101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:51:56.846550 7101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data/instance:
uuid: "2a563337cda04b02bde7e5909c712343"
format_stamp: "Formatted at 2026-09-27 22:51:56 on dist-test-slave-df53"
server_key: "3a540e9787a7e9f45e2ceebd6928220a"
server_key_iv: "f8f1059b11ae321d209ca39ec8ccc8cb"
server_key_version: "encryptionkey@0"
I20260927 22:51:56.847280 7101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal/instance:
uuid: "2a563337cda04b02bde7e5909c712343"
format_stamp: "Formatted at 2026-09-27 22:51:56 on dist-test-slave-df53"
server_key: "3a540e9787a7e9f45e2ceebd6928220a"
server_key_iv: "f8f1059b11ae321d209ca39ec8ccc8cb"
server_key_version: "encryptionkey@0"
I20260927 22:51:56.852056 7101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:51:56.855226 7203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:56.856491 7101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:51:56.856670 7101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal
uuid: "2a563337cda04b02bde7e5909c712343"
format_stamp: "Formatted at 2026-09-27 22:51:56 on dist-test-slave-df53"
server_key: "3a540e9787a7e9f45e2ceebd6928220a"
server_key_iv: "f8f1059b11ae321d209ca39ec8ccc8cb"
server_key_version: "encryptionkey@0"
I20260927 22:51:56.856846 7101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:56.883809 7101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:56.887661 7101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:56.887933 7101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:56.907346 7101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46589
I20260927 22:51:56.907349 7254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46589 every 8 connection(s)
I20260927 22:51:56.908813 7101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:51:56.911934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 7101
I20260927 22:51:56.912763 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:56.922807 7255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:51:56.937918 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.914108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.97.60:46589)
0927 22:51:56.914971 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.914981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.915019 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:51:56.917549 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.917570 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.918025 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.918452 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.920682 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.921988 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.923737 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.924408 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.924548 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.929086 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.929100 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.929112 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.929150 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.933509 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.934315 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.934321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.934326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.934514 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.934997 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.935006 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.935009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.935559 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.935833 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.936152 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.936558 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":411,"threads_started":1}
I20260927 22:51:56.941121 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.925559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45278)
0927 22:51:56.926320 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.926363 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:56.926511 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:56.926895 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:56.926899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:56.927466 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:56.927685 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.927692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.928736 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.928741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.930424 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.930439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.931395 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.931403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.931579 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.932201 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:56.932223 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:56.935280 (+ 3057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:56.938787 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.938795 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.938799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.939295 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.939663 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.939667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.939670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.939800 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.940321 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:56.940332 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:56.940490 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 22:51:56.940676 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 22:51:56.940870 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":353,"threads_started":1}
I20260927 22:51:56.941167 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.925731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.0.97.62:35383)
0927 22:51:56.925926 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.925934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.926465 (+ 531us) server_negotiation.cc:407] Connection header received
0927 22:51:56.926635 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.926643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.926727 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.926841 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.927827 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.928598 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.930588 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.931211 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.931311 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.935451 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.935457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.935461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.935497 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.938670 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.939425 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.939429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.939431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.939546 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.939901 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.939904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.939906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.940171 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.940322 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.940790 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.940949 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:51:56.942514 7255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 22:51:56.954161 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.943139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.0.1:52894)
0927 22:51:56.943304 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.943322 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:56.943417 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:56.943665 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:56.943668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:56.943951 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:56.944091 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.944098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.945037 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.945041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.945993 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.946003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.946611 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.946615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.946739 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.947238 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:56.947254 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:56.948962 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:56.951935 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.951942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.951945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.952307 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.952668 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.952674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.952678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.952801 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.953268 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:56.953273 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:56.953410 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 22:51:56.953552 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:51:56.953847 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:51:56.954188 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.943282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.97.61:38543)
0927 22:51:56.943428 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.943432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.943451 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:51:56.943505 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.943508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.943561 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.943638 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.944220 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.944912 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.946111 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.946497 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.946586 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.949149 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.949153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.949157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.949185 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.951814 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.952421 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.952428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.952431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.952562 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.952904 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.952911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.952914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.953158 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.953302 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.953670 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.953871 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:51:56.955168 7255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:51:56 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 22:51:56.966630 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.955827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46589 (local address 127.0.0.1:51352)
0927 22:51:56.955962 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:56.955975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:56.956046 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:56.956349 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:56.956352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:56.956653 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:56.956771 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.956778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.957726 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.957734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.958501 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:56.958510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.959212 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.959218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:56.959399 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.959888 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:56.959902 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:56.961875 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:56.964413 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.964420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.964425 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.964747 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.965083 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:56.965089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:56.965092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:56.965236 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:56.965742 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:56.965747 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:56.965883 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 22:51:56.966011 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 22:51:56.966259 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 22:51:56.966706 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:56.955976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.0.97.60:46589)
0927 22:51:56.956121 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:56.956125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:56.956139 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:51:56.956183 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:56.956186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:56.956232 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:56.956305 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:56.956895 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.957572 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.958672 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:56.959066 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:56.959200 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:56.962027 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:56.962037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:56.962039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:56.962063 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:56.964296 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.964860 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.964865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.964868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.964976 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:56.965331 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:56.965339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:56.965341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:56.965568 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:56.965775 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:56.966094 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:51:56.966395 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":58}
I20260927 22:51:56.971128 7255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Bootstrap starting.
I20260927 22:51:56.974591 7255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:56.975549 7255 log.cc:824] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:56.978112 7255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: No bootstrap required, opened a new log
I20260927 22:51:56.984728 7255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } }
I20260927 22:51:56.985033 7255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:56.985165 7255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a563337cda04b02bde7e5909c712343, State: Initialized, Role: FOLLOWER
I20260927 22:51:56.985736 7255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } }
I20260927 22:51:56.986649 7261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } } }
I20260927 22:51:56.987004 7261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:51:56.987645 7255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:51:56.997054 7272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:51:56.997200 7272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:51:57.000763 7101 master_runner.cc:428] Detected that this master 2a563337cda04b02bde7e5909c712343 is joining an existing cluster
I20260927 22:51:57.000882 7101 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:46589
I20260927 22:51:57.022027 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.004759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.0.97.62:35383)
0927 22:51:57.004943 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.004949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.006423 (+ 1474us) server_negotiation.cc:407] Connection header received
0927 22:51:57.006682 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.006688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.006763 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.006833 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.008284 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.009460 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.011683 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.012345 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.012451 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.015077 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.015085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.015091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.015131 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.018812 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.019610 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.019618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.019624 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.019789 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.020265 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.020270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.020273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.020614 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.020752 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.021462 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.021704 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 22:51:57.023038 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.004508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45282)
0927 22:51:57.006331 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.006351 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:57.006524 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:57.007042 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:57.007047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:57.007705 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:57.008113 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.008126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.009687 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.009696 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.011476 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.011491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.012794 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.012801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.012965 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.013666 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:57.013689 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:57.014872 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:57.018960 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.018965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.018969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.019453 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.019941 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.019949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.019952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.020135 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.020832 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:57.020837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:57.021135 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 22:51:57.022260 (+ 1125us) client_negotiation.cc:240] Negotiation successful
0927 22:51:57.022574 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1695,"spinlock_wait_cycles":26880,"thread_start_us":1533,"threads_started":1}
I20260927 22:51:57.028308 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.004517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.0.97.61:38543)
0927 22:51:57.004712 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.004718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.010079 (+ 5361us) server_negotiation.cc:407] Connection header received
0927 22:51:57.010618 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.010626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.010751 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.010849 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.012707 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.014114 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.018025 (+ 3911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.018680 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.018788 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.021298 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.021307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.021313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.021351 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.025194 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.026022 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.026028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.026031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.026183 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.026641 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.026649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.026655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.026963 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.027115 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.027744 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.027959 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:51:57.029469 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.005190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.0.1:52898)
0927 22:51:57.009954 (+ 4764us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.009975 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:57.010407 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:57.011102 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:57.011108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:57.012054 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:57.012468 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.012495 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.014467 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.014485 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.017334 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.017353 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.019101 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.019109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.019339 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.019972 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:57.020011 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:57.021096 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:57.025348 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.025352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.025357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.025866 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.026331 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.026337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.026342 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.026508 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.027198 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:57.027205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:57.027426 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 22:51:57.028582 (+ 1156us) client_negotiation.cc:240] Negotiation successful
0927 22:51:57.029001 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4587,"mutex_wait_us":5,"thread_start_us":12403,"threads_started":1}
I20260927 22:51:57.033221 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.005904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46589 (local address 127.0.0.1:51362)
0927 22:51:57.017908 (+ 12004us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.017924 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:57.018071 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:57.018439 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:57.018443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:57.019068 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:57.019397 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.019408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.020968 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.020975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.022680 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.022696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.023992 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.023999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.024168 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.024746 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:57.024805 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:57.026259 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:57.030426 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.030430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.030433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.030899 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.031293 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.031298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.031301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.031453 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.032129 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:57.032135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:57.032300 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 22:51:57.032447 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 22:51:57.032724 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11863,"mutex_wait_us":2,"thread_start_us":4198,"threads_started":1}
I20260927 22:51:57.033581 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.005010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.0.97.60:46589)
0927 22:51:57.005230 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.005238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.017984 (+ 12746us) server_negotiation.cc:407] Connection header received
0927 22:51:57.018172 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.018177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.018228 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.018280 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.019575 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.020739 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.022831 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.023588 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.023691 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.026429 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.026437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.026443 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.026472 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.030299 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.031034 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.031040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.031044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.031175 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.031594 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.031601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.031609 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.031972 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.032146 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.032572 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.032901 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":31,"spinlock_wait_cycles":23296}
Sep 27 22:51:57 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549514, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 22:51:57.043445 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 6602
I20260927 22:51:57.043507 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.028561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46589 (local address 127.0.0.1:51378)
0927 22:51:57.028953 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.028969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:57.029059 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:57.030720 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:57.030725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:57.031127 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:57.031325 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.031340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.032986 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.032992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.034233 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.034244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.035539 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.035545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.035720 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.036304 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:57.036320 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:57.038223 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:57.041164 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.041169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.041173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.041505 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.041869 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.041874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.041877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.042010 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.042595 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:57.042601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:57.042750 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 22:51:57.042904 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:51:57.043147 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260927 22:51:57.043598 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.029241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.0.97.60:46589)
0927 22:51:57.030350 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.030358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.030379 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:51:57.030456 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.030463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.030545 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.031308 (+ 763us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.031643 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.032815 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.034536 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.035376 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.035516 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.038416 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.038421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.038424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.038448 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.040992 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.041637 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.041644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.041647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.041758 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.042118 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.042124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.042128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.042465 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.042621 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.043002 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.043193 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":1428,"threads_started":1}
I20260927 22:51:57.044762 6541 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:46589: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2a563337cda04b02bde7e5909c712343"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 46589
}
attrs {
promote: true
}
}
dest_uuid: "6be24da53c1245a8afab32e4c229cc0f"
cas_config_opid_index: 5
I20260927 22:51:57.045570 6541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } }
I20260927 22:51:57.047168 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2a563337cda04b02bde7e5909c712343 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:57.050797 7099 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:46589
I20260927 22:51:57.051950 7230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:57.056262 7230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 FOLLOWER]: Refusing update from remote peer 6be24da53c1245a8afab32e4c229cc0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
Sep 27 22:51:57 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 22:51:57.057437 7101 master.cc:537] Master@127.0.97.60:46589 shutting down...
I20260927 22:51:57.071038 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.045522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.0.97.1:44033)
0927 22:51:57.046231 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.046239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.046268 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:51:57.046346 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.046354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.046609 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.048355 (+ 1746us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.048558 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.050067 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.051847 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.052728 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.059232 (+ 6504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.059559 (+ 327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.059573 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.059594 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.059634 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.065416 (+ 5782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.066702 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.066715 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.066721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.066894 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.067729 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.067737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.067742 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.068219 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.069928 (+ 1709us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.070130 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.070456 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":369,"threads_started":1}
I20260927 22:51:57.072559 6725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:57.074551 7285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:57.077808 7148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 LEARNER]: Refusing update from remote peer 6be24da53c1245a8afab32e4c229cc0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 22:51:57.078744 7283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 22:51:57.085371 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:57.085875 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:57.088864 7148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2a563337cda04b02bde7e5909c712343 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:57.096273 7101 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:57.096750 7101 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:57.096978 7101 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: stopping tablet replica
I20260927 22:51:57.099179 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: true } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:57.099522 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: This master's current role is: LEARNER
W20260927 22:51:57.102281 6528 connection.cc:535] client connection to 127.0.97.60:46589 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 22:51:57.102427 6528 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f -> Peer 2a563337cda04b02bde7e5909c712343 (127.0.97.60:46589): Couldn't send request to peer 2a563337cda04b02bde7e5909c712343. Status: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:51:57.103181 7283 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f: attempting to promote NON_VOTER 2a73ae77b7f54c929293c60d37f40189 to VOTER
I20260927 22:51:57.104158 7283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } }
I20260927 22:51:57.107498 7148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 LEARNER]: Refusing update from remote peer 6be24da53c1245a8afab32e4c229cc0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 22:51:57.108018 7285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 22:51:57.108820 7101 master.cc:552] Master@127.0.97.60:46589 shutdown complete.
I20260927 22:51:57.108958 7101 master_runner.cc:305] Clearing existing system tablet
I20260927 22:51:57.109395 7101 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:57.109512 7101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:57.111912 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2a73ae77b7f54c929293c60d37f40189 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:57.112389 7148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2a73ae77b7f54c929293c60d37f40189 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:57.115175 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.114725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46589 (local address 127.0.0.1:51392)
0927 22:51:57.114901 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.114978 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:51:57.117120 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:57.117329 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:51:57.117532 7099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:57.117797 7099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:57.118487 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.117692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46589 (local address 127.0.0.1:51402)
0927 22:51:57.118263 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.118339 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474}
I20260927 22:51:57.118499 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
W20260927 22:51:57.118852 6528 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f -> Peer 2a563337cda04b02bde7e5909c712343 (127.0.97.60:46589): Couldn't send request to peer 2a563337cda04b02bde7e5909c712343. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:51:57.118850 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:57.119875 7101 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.000s
I20260927 22:51:57.121492 7292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:57.121827 7101 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:57.122089 7101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 22:51:57.122191 7101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal
uuid: "2a563337cda04b02bde7e5909c712343"
format_stamp: "Formatted at 2026-09-27 22:51:56 on dist-test-slave-df53"
server_key: "3a540e9787a7e9f45e2ceebd6928220a"
server_key_iv: "f8f1059b11ae321d209ca39ec8ccc8cb"
server_key_version: "encryptionkey@0"
I20260927 22:51:57.127014 7101 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:57.133631 7101 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:51:57.133764 7101 log.cc:1196] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 22:51:57.134262 7101 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Deleting consensus metadata
I20260927 22:51:57.134496 7101 master_runner.cc:315] Copying system tablet from 127.0.97.62:35383
I20260927 22:51:57.139034 7101 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:35383
I20260927 22:51:57.155577 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.139816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.0.97.62:35383)
0927 22:51:57.140003 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.140010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.142123 (+ 2113us) server_negotiation.cc:407] Connection header received
0927 22:51:57.142291 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.142298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.142378 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.142484 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.143422 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.144604 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.146511 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.147153 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.147290 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.149458 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.149465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.149470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.149512 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.152821 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.153564 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.153570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.153574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.153735 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.154140 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.154145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.154148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.154462 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.154630 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.155002 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.155221 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:51:57.155601 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.139741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45298)
0927 22:51:57.141995 (+ 2254us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.142013 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:57.142167 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:57.142520 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:57.142525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:57.143007 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:57.143272 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.143285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.144759 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.144768 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.146343 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.146360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.147339 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.147345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.147529 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.148205 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:57.148229 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:57.149232 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:57.152961 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.152967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.152971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.153446 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.153856 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.153860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.153864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.154023 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.154593 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:57.154598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:57.154756 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 22:51:57.154897 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:51:57.155183 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2104,"thread_start_us":1986,"threads_started":1}
I20260927 22:51:57.156466 6561 tablet_copy_service.cc:140] P 6be24da53c1245a8afab32e4c229cc0f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2a563337cda04b02bde7e5909c712343 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:45298)
I20260927 22:51:57.156635 6561 tablet_copy_service.cc:161] P 6be24da53c1245a8afab32e4c229cc0f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2a563337cda04b02bde7e5909c712343 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:45298: session id = 2a563337cda04b02bde7e5909c712343-00000000000000000000000000000000
I20260927 22:51:57.160236 6561 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:51:57.161981 7101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:51:57.168987 7101 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: tablet copy: Starting download of 0 data blocks...
I20260927 22:51:57.169226 7101 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: tablet copy: Starting download of 1 WAL segments...
I20260927 22:51:57.171382 7101 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:51:57.175802 6561 tablet_copy_service.cc:342] P 6be24da53c1245a8afab32e4c229cc0f: Request end of tablet copy session 2a563337cda04b02bde7e5909c712343-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:45298
I20260927 22:51:57.175923 6561 tablet_copy_service.cc:433] P 6be24da53c1245a8afab32e4c229cc0f: ending tablet copy session 2a563337cda04b02bde7e5909c712343-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2a563337cda04b02bde7e5909c712343
I20260927 22:51:57.187568 7101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:57.193459 7300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:57.196735 7299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:57.200582 7302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:57.201063 7101 server_base.cc:1034] running on GCE node
I20260927 22:51:57.201404 7101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:57.202108 7101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:57.203578 7101 hybrid_clock.cc:648] HybridClock initialized: now 1790549517203387 us; error 29 us; skew 500 ppm
I20260927 22:51:57.203860 7101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:51:57.205060 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 6602
Sep 27 22:51:57 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:57.210189 7101 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 22:51:57.219048 7101 webserver.cc:460] Webserver started at http://127.0.97.60:39569/ using document root <none> and password file <none>
I20260927 22:51:57.219359 7101 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:57.219468 7101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:57.222457 7101 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:51:57.223132 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44033
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35383
--webserver_port=33035
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35383,127.0.97.61:38543
--tserver_master_addrs=127.0.97.62:35383,127.0.97.61:38543,127.0.97.60:46589 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:51:57.224116 7309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:57.224537 7101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 22:51:57.224659 7101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal
uuid: "2a563337cda04b02bde7e5909c712343"
format_stamp: "Formatted at 2026-09-27 22:51:56 on dist-test-slave-df53"
server_key: "3a540e9787a7e9f45e2ceebd6928220a"
server_key_iv: "f8f1059b11ae321d209ca39ec8ccc8cb"
server_key_version: "encryptionkey@0"
I20260927 22:51:57.224838 7101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:57.236267 7101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:57.239565 7101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:57.282323 7101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46589
I20260927 22:51:57.282357 7361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46589 every 8 connection(s)
I20260927 22:51:57.283298 7101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:51:57.290526 7362 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:51:57.294845 7362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Bootstrap starting.
I20260927 22:51:57.309741 7362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:51:57.310166 7362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Bootstrap complete.
I20260927 22:51:57.311040 7362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } }
I20260927 22:51:57.311282 7362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a563337cda04b02bde7e5909c712343, State: Initialized, Role: LEARNER
I20260927 22:51:57.311595 7362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } }
I20260927 22:51:57.312431 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:57.312811 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:51:57.312819 7362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:51:57.323148 7376 catalog_manager.cc:1259] Loaded cluster ID: a5c4f88810be476fa78002bf583faa9c
I20260927 22:51:57.323228 7376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: loading cluster ID for follower catalog manager: success
I20260927 22:51:57.326553 7376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: acquiring CA information for follower catalog manager: success
I20260927 22:51:57.328570 7376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:51:57.578240 7314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:57.578650 7314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:57.578804 7314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:57.586182 7314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:57.586304 7314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:57.586469 7314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:51:57.597077 7314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33035
--tserver_master_addrs=127.0.97.62:35383,127.0.97.61:38543,127.0.97.60:46589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:57.598420 7314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:57.599726 7314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:57.613715 7382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:57.613947 7384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:57.613832 7381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:57.614866 7314 server_base.cc:1034] running on GCE node
I20260927 22:51:57.615515 7314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:57.616389 7314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:57.617640 7314 hybrid_clock.cc:648] HybridClock initialized: now 1790549517617568 us; error 64 us; skew 500 ppm
I20260927 22:51:57.617962 7314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:57 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:57.620999 7314 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:51:57.622376 7314 webserver.cc:460] Webserver started at http://127.0.97.1:33035/ using document root <none> and password file <none>
I20260927 22:51:57.622991 7314 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:57.623149 7314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:57.628016 7314 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:51:57.630489 7391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:57.631664 7314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:51:57.631829 7314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "820c3e17809a41d2847c3da4760d7aa0"
format_stamp: "Formatted at 2026-09-27 22:51:54 on dist-test-slave-df53"
server_key: "ecc38c6bb0ba55f5a0a7265de509c6d3"
server_key_iv: "f0e354985f3362be3ecb26e62cc145ff"
server_key_version: "encryptionkey@0"
I20260927 22:51:57.632792 7314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:57.643366 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.637261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46589 (local address 127.0.0.1:51410)
0927 22:51:57.637827 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.637851 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:57.638018 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:57.638466 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:57.638471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:57.638503 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:57.638710 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.638719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.640346 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.640351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.641477 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.641488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.642573 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.642579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.642777 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.642814 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:51:57.642919 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:51:57.643058 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":260,"threads_started":1}
I20260927 22:51:57.643386 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.637356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.0.97.60:46589)
0927 22:51:57.638091 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.638140 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.638164 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:51:57.638235 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.638239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.638301 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.638446 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:57.638847 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.640196 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.641647 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.642401 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.642563 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.642751 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.642940 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.643060 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":467,"threads_started":1}
I20260927 22:51:57.644256 7337 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 22:51:57.644745 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6be24da53c1245a8afab32e4c229cc0f. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:57.644989 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:51:57.663117 7314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:57.666801 7314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:57.667073 7314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:57.667951 7314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:57.669193 7314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:57.669297 7314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:57.669421 7314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:57.669518 7314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:57.702864 7314 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44033
I20260927 22:51:57.702945 7505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44033 every 8 connection(s)
I20260927 22:51:57.704253 7314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:51:57.706897 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 7314
I20260927 22:51:57.707232 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 6736
Sep 27 22:51:57 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 22:51:57 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 22:51:57 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 22:51:57 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 22:51:57.747388 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.708889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60725 (local address 127.0.97.61:38543)
0927 22:51:57.722037 (+ 13148us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.722055 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.724771 (+ 2716us) server_negotiation.cc:407] Connection header received
0927 22:51:57.726288 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.726293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.726381 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.726475 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.729108 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.730377 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.732889 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.733989 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.734089 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.738600 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.738610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.738613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.738651 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.743648 (+ 4997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.744712 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.744717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.744721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.744852 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.745361 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.745366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.745370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.745757 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.745867 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.746796 (+ 929us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.746997 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12991,"thread_start_us":12885,"threads_started":1}
I20260927 22:51:57.747388 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.707579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54829 (local address 127.0.97.62:35383)
0927 22:51:57.713936 (+ 6357us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.713942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.723113 (+ 9171us) server_negotiation.cc:407] Connection header received
0927 22:51:57.726216 (+ 3103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.726223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.726315 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.726394 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.729114 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.730242 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.732890 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.734009 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.734103 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.739540 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.739547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.739551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.739593 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.744306 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.745130 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.745135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.745138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.745277 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.745700 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.745705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.745709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.746028 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.746143 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.746795 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.746997 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6235,"thread_start_us":6138,"threads_started":1}
I20260927 22:51:57.747391 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.714402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49739 (local address 127.0.97.60:46589)
0927 22:51:57.714587 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.714594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.725142 (+ 10548us) server_negotiation.cc:407] Connection header received
0927 22:51:57.726276 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.726281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.726354 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.726421 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.729103 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.730157 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.732903 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.733558 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.733722 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.740137 (+ 6415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.740144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.740149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.740189 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.743845 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.744842 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.744847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.744850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.744979 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.745453 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.745458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.745463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.745823 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.745930 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.746786 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.747023 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:51:57.749547 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.708599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.0.97.2:35461)
0927 22:51:57.723572 (+ 14973us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:57.723583 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:57.725670 (+ 2087us) server_negotiation.cc:407] Connection header received
0927 22:51:57.725872 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:57.725881 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:57.726176 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:57.726321 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:57.728589 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.730225 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.732123 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.732926 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.733037 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.741303 (+ 8266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:57.741318 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:57.741335 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:57.741406 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:57.745808 (+ 4402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.746633 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.746643 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.746647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.746788 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:57.747208 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:57.747214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:57.747218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:57.747642 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:57.747893 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:57.748957 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0927 22:51:57.749192 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14800,"thread_start_us":14660,"threads_started":1}
I20260927 22:51:57.750679 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.708874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.97.1:60725)
0927 22:51:57.724064 (+ 15190us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.724082 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:57.725926 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:57.726861 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:57.726873 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:57.727844 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:57.728917 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.728939 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.730671 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.730677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.732747 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.732756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.734437 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.734441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.734706 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.735563 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:57.735587 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:57.738376 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:57.743811 (+ 5435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.743822 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.743840 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.744574 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.744996 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.745000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.745003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.745230 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.745973 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:57.745981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:57.746379 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 22:51:57.747604 (+ 1225us) client_negotiation.cc:240] Negotiation successful
0927 22:51:57.748270 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15027,"mutex_wait_us":494,"spinlock_wait_cycles":50816,"thread_start_us":14860,"threads_started":1}
I20260927 22:51:57.750751 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.708560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.1:54829)
0927 22:51:57.722646 (+ 14086us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.722690 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:57.725922 (+ 3232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:57.726861 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:57.726873 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:57.727844 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:57.728931 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.728939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.730426 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.730431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.732732 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.732747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.734420 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.734427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.734706 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.735349 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:57.735396 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:57.739052 (+ 3656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:57.744483 (+ 5431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.744489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.744494 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.744990 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.745412 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.745418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.745421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.745549 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.746225 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:57.746229 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:57.746379 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 22:51:57.747617 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0927 22:51:57.748388 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13799,"mutex_wait_us":679,"spinlock_wait_cycles":10624,"thread_start_us":16085,"threads_started":1}
I20260927 22:51:57.750941 6859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:57.750715 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:57.714357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46589 (local address 127.0.97.1:49739)
0927 22:51:57.725025 (+ 10668us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:57.725044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:57.725922 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:57.726865 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:57.726874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:57.727844 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:57.728927 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.728940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.730823 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.730829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.732747 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:57.732761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:57.734095 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:57.734103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:57.734435 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:57.735190 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:57.735230 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:57.739949 (+ 4719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:57.744025 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.744032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.744038 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.744699 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.745140 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:57.745145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:57.745148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:57.745323 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:57.746020 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:57.746024 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:57.746381 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 22:51:57.748201 (+ 1820us) client_negotiation.cc:240] Negotiation successful
0927 22:51:57.748522 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10539,"mutex_wait_us":271,"spinlock_wait_cycles":32768,"thread_start_us":7898,"threads_started":1}
I20260927 22:51:57.752905 7508 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38543
I20260927 22:51:57.752945 7507 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35383
I20260927 22:51:57.753211 7508 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:57.753218 7507 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:57.754257 7508 heartbeater.cc:511] Master 127.0.97.61:38543 requested a full tablet report, sending...
I20260927 22:51:57.754297 7507 heartbeater.cc:511] Master 127.0.97.62:35383 requested a full tablet report, sending...
I20260927 22:51:57.756095 7506 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46589
I20260927 22:51:57.756131 6541 ts_manager.cc:194] Re-registered known tserver with Master: 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033)
I20260927 22:51:57.756516 7138 ts_manager.cc:194] Registered new tserver with Master: 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033)
I20260927 22:51:57.756645 7506 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:57.756987 7506 heartbeater.cc:511] Master 127.0.97.60:46589 requested a full tablet report, sending...
I20260927 22:51:57.757616 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54829
I20260927 22:51:57.758613 7327 ts_manager.cc:194] Registered new tserver with Master: 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033)
I20260927 22:51:57.838371 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 6736
I20260927 22:51:57.854095 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35461
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35383
--webserver_port=36919
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35383,127.0.97.61:38543
--tserver_master_addrs=127.0.97.62:35383,127.0.97.61:38543,127.0.97.60:46589 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:58.166594 7518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:58.166832 7518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:58.166996 7518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:51:58.177759 7518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:58.177877 7518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:58.178045 7518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:51:58.188232 7518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36919
--tserver_master_addrs=127.0.97.62:35383,127.0.97.61:38543,127.0.97.60:46589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:58.189345 7518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:58.190496 7518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:58.202718 7523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:58.202776 7526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:58.202776 7524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:58.203585 7518 server_base.cc:1034] running on GCE node
I20260927 22:51:58.204123 7518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:58.204900 7518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:58.206118 7518 hybrid_clock.cc:648] HybridClock initialized: now 1790549518206066 us; error 58 us; skew 500 ppm
I20260927 22:51:58.206406 7518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:58 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:58.209509 7518 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:51:58.210844 7518 webserver.cc:460] Webserver started at http://127.0.97.2:36919/ using document root <none> and password file <none>
I20260927 22:51:58.211555 7518 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:58.211719 7518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:58.217067 7518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:51:58.218194 7099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: true } } }
I20260927 22:51:58.218528 7099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:58.220031 7099 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f: attempting to promote NON_VOTER 2a563337cda04b02bde7e5909c712343 to VOTER
I20260927 22:51:58.220362 7536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.220659 7099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } }
I20260927 22:51:58.221838 7518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:51:58.222640 7337 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 LEARNER]: Refusing update from remote peer 6be24da53c1245a8afab32e4c229cc0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:51:58.222687 7148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Refusing update from remote peer 6be24da53c1245a8afab32e4c229cc0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:51:58.221993 7518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "400068953a8d411086ce260244bbdf2d"
format_stamp: "Formatted at 2026-09-27 22:51:55 on dist-test-slave-df53"
server_key: "164dac1355f3f6c6858b3a4970917b73"
server_key_iv: "6165a05d73972ebee7e09a736082b2ef"
server_key_version: "encryptionkey@0"
I20260927 22:51:58.223207 7533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:51:58.223222 7518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:58.223587 7534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:51:58.226430 7533 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a563337cda04b02bde7e5909c712343 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:51:58.227100 7148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a563337cda04b02bde7e5909c712343 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:51:58.230211 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:51:58.230734 7534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:51:58.230538 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:58.230975 7534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:58.231717 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:51:58.231958 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f [sys.catalog]: This master's current role is: LEADER
I20260927 22:51:58.232290 7337 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a563337cda04b02bde7e5909c712343 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:51:58.232545 7544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:51:58.232939 7544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:51:58.235522 7545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6be24da53c1245a8afab32e4c229cc0f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:51:58.235816 7545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:51:58.248813 7518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:58.252729 7518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:58.252995 7518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:58.253904 7518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:58.255182 7518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:58.255303 7518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.255424 7518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:58.255508 7518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.288488 7518 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35461
I20260927 22:51:58.288537 7653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35461 every 8 connection(s)
I20260927 22:51:58.289929 7518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:51:58.291725 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 7518
I20260927 22:51:58.292073 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 6869
Sep 27 22:51:58 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 22:51:58 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 22:51:58 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 22:51:58.335237 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.296824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50737 (local address 127.0.97.61:38543)
0927 22:51:58.310004 (+ 13180us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.310013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.310730 (+ 717us) server_negotiation.cc:407] Connection header received
0927 22:51:58.310805 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.310810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.310869 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.310985 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.314048 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.315604 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.320158 (+ 4554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.320955 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.321069 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.326428 (+ 5359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.326435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.326443 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.326498 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.330887 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.331946 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.331954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.331960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.332092 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.333273 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.333280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.333283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.333759 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.333947 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.334647 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.334876 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13029,"thread_start_us":12867,"threads_started":1}
I20260927 22:51:58.335716 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.292633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37381 (local address 127.0.97.62:35383)
0927 22:51:58.315620 (+ 22987us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.315629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.315654 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:51:58.315901 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.315909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.316029 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.316148 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.317619 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.319177 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.321836 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.322740 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.322855 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.328325 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.328330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.328337 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.328386 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.332262 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.333020 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.333026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.333029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.333256 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.333649 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.333654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.333664 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.333989 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.334127 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.334901 (+ 774us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.335221 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22820,"thread_start_us":23045,"threads_started":1}
I20260927 22:51:58.336848 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.296260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.97.2:50737)
0927 22:51:58.310410 (+ 14150us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.310427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.310544 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.311428 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.311444 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.312680 (+ 1236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:58.313809 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.313838 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.315837 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.315845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.319966 (+ 4121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.319984 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.321456 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.321465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.321827 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.322665 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:58.322703 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:58.326152 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:58.331044 (+ 4892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.331054 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.331070 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.331799 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.332262 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.332267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.332270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.333112 (+ 842us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.333879 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:58.333886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:58.334283 (+ 397us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.334554 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.335078 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14032,"mutex_wait_us":394,"thread_start_us":14209,"threads_started":1}
I20260927 22:51:58.337129 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.293123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:37381)
0927 22:51:58.306331 (+ 13208us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.306394 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.309367 (+ 2973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.316354 (+ 6987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.316362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.316949 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:58.317391 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.317403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.319470 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.319482 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.321639 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.321657 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.323226 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.323234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.323529 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.325261 (+ 1732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:58.325289 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:58.328090 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:58.332394 (+ 4304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.332398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.332401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.332882 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.333377 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.333383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.333387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.333516 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.334151 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:58.334156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:58.334301 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.334555 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.336161 (+ 1606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12844,"mutex_wait_us":1220,"spinlock_wait_cycles":27264,"thread_start_us":12611,"threads_started":1}
Sep 27 22:51:58 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 22:51:58.338550 7655 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35383
I20260927 22:51:58.338855 7655 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:58.339560 7655 heartbeater.cc:511] Master 127.0.97.62:35383 requested a full tablet report, sending...
I20260927 22:51:58.340243 7656 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38543
I20260927 22:51:58.340376 7656 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:58.340802 7656 heartbeater.cc:511] Master 127.0.97.61:38543 requested a full tablet report, sending...
I20260927 22:51:58.341068 6541 ts_manager.cc:194] Re-registered known tserver with Master: 400068953a8d411086ce260244bbdf2d (127.0.97.2:35461)
I20260927 22:51:58.342345 7138 ts_manager.cc:194] Registered new tserver with Master: 400068953a8d411086ce260244bbdf2d (127.0.97.2:35461)
I20260927 22:51:58.342345 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37381
I20260927 22:51:58.345974 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.311179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46589 (local address 127.0.97.2:36123)
0927 22:51:58.323213 (+ 12034us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.323231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.324406 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.324988 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.324994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.325655 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:58.326024 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.326038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.327495 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.327506 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.329082 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.329097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.330342 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.330349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.330518 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.331043 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:58.331062 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:58.339309 (+ 8247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:58.343511 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.343514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.343517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.343847 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.344223 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.344229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.344233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.344367 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.345039 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:58.345042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:58.345151 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.345293 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.345573 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11879,"thread_start_us":12150,"threads_started":1}
I20260927 22:51:58.345986 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.311207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36123 (local address 127.0.97.60:46589)
0927 22:51:58.317550 (+ 6343us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.317557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.323314 (+ 5757us) server_negotiation.cc:407] Connection header received
0927 22:51:58.324581 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.324593 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.324672 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.324769 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.326213 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.327314 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.329238 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.329973 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.330066 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.339843 (+ 9777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.339851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.339855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.339894 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.343374 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.343971 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.343976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.343979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.344113 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.344478 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.344483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.344487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.344841 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.344989 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.345396 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.345585 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6199,"thread_start_us":6075,"threads_started":1}
I20260927 22:51:58.346469 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.293485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.0.97.3:35043)
0927 22:51:58.324283 (+ 30798us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.324293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.324321 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:51:58.324388 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.324399 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.324700 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.324916 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.326930 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.329458 (+ 2528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.332104 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.332827 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.332968 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.338722 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.338735 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.338749 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.338786 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.343568 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.344372 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.344380 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.344383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.344528 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.344930 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.344933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.344935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.345205 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.345483 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.345814 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.346054 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30520,"thread_start_us":31107,"threads_started":1}
I20260927 22:51:58.346871 7654 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46589
I20260927 22:51:58.347003 7654 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:58.347312 7654 heartbeater.cc:511] Master 127.0.97.60:46589 requested a full tablet report, sending...
I20260927 22:51:58.347740 6992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:58.348793 7327 ts_manager.cc:194] Registered new tserver with Master: 400068953a8d411086ce260244bbdf2d (127.0.97.2:35461)
I20260927 22:51:58.429971 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 6869
I20260927 22:51:58.446435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35043
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35383
--webserver_port=32821
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:35383,127.0.97.61:38543
--tserver_master_addrs=127.0.97.62:35383,127.0.97.61:38543,127.0.97.60:46589 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:51:58.758368 7667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:51:58.758669 7667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:51:58.758844 7667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 22:51:58.759996 7507 heartbeater.cc:503] Master 127.0.97.62:35383 was elected leader, sending a full tablet report...
W20260927 22:51:58.766671 7667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:51:58.766770 7667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:51:58.766878 7667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:51:58.776203 7667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32821
--tserver_master_addrs=127.0.97.62:35383,127.0.97.61:38543,127.0.97.60:46589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:51:58.777383 7667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:51:58.778637 7667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:51:58.791493 7673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:58.791611 7672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:51:58.791584 7675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:51:58.792429 7667 server_base.cc:1034] running on GCE node
I20260927 22:51:58.792958 7667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:51:58.793836 7667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:51:58.795040 7667 hybrid_clock.cc:648] HybridClock initialized: now 1790549518795005 us; error 46 us; skew 500 ppm
I20260927 22:51:58.795343 7667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:51:58 dist-test-slave-df53 krb5kdc[6496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:51:58.798337 7667 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:51:58.799683 7667 webserver.cc:460] Webserver started at http://127.0.97.3:32821/ using document root <none> and password file <none>
I20260927 22:51:58.800328 7667 fs_manager.cc:359] Metadata directory not provided
I20260927 22:51:58.800482 7667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:51:58.805439 7667 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:51:58.808133 7682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.809485 7667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:51:58.809685 7667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "53f009fcb18843b9bebd0475077d28f6"
format_stamp: "Formatted at 2026-09-27 22:51:55 on dist-test-slave-df53"
server_key: "b296102a58073335e322a6be619fcbac"
server_key_iv: "6a95d9520638522ad5947519f35d1e6e"
server_key_version: "encryptionkey@0"
I20260927 22:51:58.810743 7667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:51:58.831559 7667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:51:58.835580 7667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:51:58.835829 7667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:51:58.836848 7667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:51:58.838282 7667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:51:58.838409 7667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.838536 7667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:51:58.838624 7667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:51:58.871101 7667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35043
I20260927 22:51:58.871165 7794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35043 every 8 connection(s)
I20260927 22:51:58.872464 7667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:51:58.875530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 7667
Sep 27 22:51:58 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 22:51:58 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 22:51:58 dist-test-slave-df53 krb5kdc[6496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 22:51:58.932034 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.877563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53413 (local address 127.0.97.62:35383)
0927 22:51:58.892826 (+ 15263us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.892836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.899513 (+ 6677us) server_negotiation.cc:407] Connection header received
0927 22:51:58.907672 (+ 8159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.907680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.907778 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.907904 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.911549 (+ 3645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.912860 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.915429 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.916075 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.916204 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.924168 (+ 7964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.924175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.924179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.924217 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.928128 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.929059 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.929067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.929074 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.929235 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.929716 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.929721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.929725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.930042 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.930168 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.931433 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.931633 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15102,"thread_start_us":15261,"threads_started":1}
I20260927 22:51:58.933254 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.877571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.3:53413)
0927 22:51:58.899408 (+ 21837us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.899429 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.907355 (+ 7926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.908179 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.908195 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.911101 (+ 2906us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:58.911390 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.911403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.913014 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.913020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.915297 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.915309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.917562 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.917569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.918372 (+ 803us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.919745 (+ 1373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:58.919775 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:58.923925 (+ 4150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:58.928256 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.928266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.928283 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.928952 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.929329 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.929335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.929339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.929577 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.930619 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:58.930627 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:58.931023 (+ 396us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.931308 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.931796 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21735,"spinlock_wait_cycles":38912,"thread_start_us":21917,"threads_started":1}
I20260927 22:51:58.934965 7796 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35383
I20260927 22:51:58.935303 7796 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:58.935755 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.901722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.0.97.60:46589)
0927 22:51:58.905653 (+ 3931us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.905662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.906794 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 22:51:58.906849 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.906853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.906911 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.907025 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.910595 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.912144 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.914344 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.915044 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.915156 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.920480 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.920488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.920493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.920527 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.926182 (+ 5655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.927107 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.927115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.927120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.927286 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.927831 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.927837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.927840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.928166 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.929261 (+ 1095us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.934273 (+ 5012us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.935137 (+ 864us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3754,"thread_start_us":3642,"threads_started":1}
I20260927 22:51:58.936215 7796 heartbeater.cc:511] Master 127.0.97.62:35383 requested a full tablet report, sending...
I20260927 22:51:58.937911 6541 ts_manager.cc:194] Re-registered known tserver with Master: 53f009fcb18843b9bebd0475077d28f6 (127.0.97.3:35043)
I20260927 22:51:58.938123 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.876600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.97.3:38341)
0927 22:51:58.899380 (+ 22780us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.899402 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.907355 (+ 7953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.908400 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.908404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.909339 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:58.910273 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.910301 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.915452 (+ 5151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.915457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.917296 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.917311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.918955 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.918962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.919154 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.919875 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:58.919899 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:58.923982 (+ 4083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:58.935058 (+ 11076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.935064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.935067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.935600 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.936092 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.936097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.936100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.936302 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.936947 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:58.936952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:58.937085 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.937230 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.937729 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22646,"thread_start_us":22964,"threads_started":1}
I20260927 22:51:58.938966 6541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53413
I20260927 22:51:58.939222 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.901656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.0.97.61:38543)
0927 22:51:58.917465 (+ 15809us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.917474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.917493 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:51:58.917534 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.917538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.917618 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.917713 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.920330 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.921491 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.923489 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.924201 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.924328 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.927826 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.927832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.927837 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.927873 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.932942 (+ 5069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.934107 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.934112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.934116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.934279 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.934708 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.934714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.934718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.937409 (+ 2691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.937688 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.938713 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.938905 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15694,"thread_start_us":15527,"threads_started":1}
I20260927 22:51:58.939920 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.875483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38341 (local address 127.0.97.61:38543)
0927 22:51:58.900287 (+ 24804us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.900291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.900307 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:51:58.908122 (+ 7815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.908130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.908251 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.908397 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.910429 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.913363 (+ 2934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.917421 (+ 4058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.918614 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.918761 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.924925 (+ 6164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.924932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.924939 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.924976 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.934917 (+ 9941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.935813 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.935818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.935822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.935960 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.936423 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.936428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.936432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.936739 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.938404 (+ 1665us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.938527 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.939527 (+ 1000us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":529,"server-negotiator.queue_time_us":24703,"thread_start_us":24560,"threads_started":1}
I20260927 22:51:58.941931 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.875962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46589 (local address 127.0.97.3:58163)
0927 22:51:58.898471 (+ 22509us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.898553 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.907741 (+ 9188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.908326 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.908330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.911814 (+ 3484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:51:58.912025 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.912034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.919525 (+ 7491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.919531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.921739 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.921753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.922909 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.922916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.923095 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.925702 (+ 2607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:51:58.925729 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:51:58.930321 (+ 4592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:51:58.934575 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.934579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.934583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.935031 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.938615 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:51:58.938621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:51:58.938625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:51:58.938805 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:51:58.939513 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:51:58.939518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:51:58.939676 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.939812 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.940142 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22123,"thread_start_us":21880,"threads_started":1}
I20260927 22:51:58.941998 7797 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38543
I20260927 22:51:58.942129 7797 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:58.942489 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.875963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58163 (local address 127.0.97.60:46589)
0927 22:51:58.899898 (+ 23935us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.899907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.899932 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:51:58.907883 (+ 7951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.907888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.907970 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.908113 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.912223 (+ 4110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.916555 (+ 4332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.921904 (+ 5349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.922557 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.922693 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.930582 (+ 7889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.930589 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.930594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.930628 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.934414 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.935209 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.935217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.935221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.935399 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.938939 (+ 3540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.938946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.938951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.939293 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.939453 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.940892 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.941093 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23790,"thread_start_us":23700,"threads_started":1}
I20260927 22:51:58.942660 7797 heartbeater.cc:511] Master 127.0.97.61:38543 requested a full tablet report, sending...
I20260927 22:51:58.943190 7795 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46589
I20260927 22:51:58.943313 7795 heartbeater.cc:464] Registering TS with master...
I20260927 22:51:58.943593 7795 heartbeater.cc:511] Master 127.0.97.60:46589 requested a full tablet report, sending...
I20260927 22:51:58.943706 7138 ts_manager.cc:194] Registered new tserver with Master: 53f009fcb18843b9bebd0475077d28f6 (127.0.97.3:35043)
I20260927 22:51:58.944505 7327 ts_manager.cc:194] Registered new tserver with Master: 53f009fcb18843b9bebd0475077d28f6 (127.0.97.3:35043)
I20260927 22:51:58.947665 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.901570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.0.97.62:35383)
0927 22:51:58.918993 (+ 17423us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.919002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.919022 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:51:58.919075 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.919082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.919161 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.919265 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:58.928178 (+ 8913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.929432 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.936926 (+ 7494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.937645 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.937769 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.941391 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:58.941397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:58.941402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:58.941436 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:58.945152 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.945832 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.945838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.945841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.945960 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:58.946307 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:58.946312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:58.946315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:58.946547 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:58.946765 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.947133 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.947353 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16575,"thread_start_us":17296,"threads_started":1}
I20260927 22:51:58.950996 6541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:51:58.957499 6541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:51:58.985039 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.974916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35043 (local address 127.0.0.1:46720)
0927 22:51:58.975930 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.975956 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.976109 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.978798 (+ 2689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.978803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.978831 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:58.979146 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.979154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.981127 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.981132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.982788 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.982802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.984227 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.984233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.984400 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.984434 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.984548 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.984676 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":729,"threads_started":1}
I20260927 22:51:58.985994 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.975525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.0.97.3:35043)
0927 22:51:58.978104 (+ 2579us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.978110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.978134 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:51:58.978205 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.978210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.978408 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.979504 (+ 1096us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:58.980004 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.980972 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.982967 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.984037 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.984195 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.984417 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.984602 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.984753 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2428,"thread_start_us":2277,"threads_started":1}
I20260927 22:51:58.990274 7730 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9bf08a489048c485292b4fb8268216 (DEFAULT_TABLE table=test-table [id=ca7b47db2c1a44589d9332d8ccb12041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:58.991931 7730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9bf08a489048c485292b4fb8268216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:58.992455 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.975525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44033 (local address 127.0.0.1:40346)
0927 22:51:58.985391 (+ 9866us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.985408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.985501 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.986370 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.986376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.986399 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:58.986527 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.986533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.988335 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.988340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.990032 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.990042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.991764 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.991771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.991955 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.991988 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.992054 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.992181 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9759,"thread_start_us":10105,"threads_started":1}
I20260927 22:51:58.993433 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.976033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.0.97.1:44033)
0927 22:51:58.985717 (+ 9684us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.985726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.985756 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:51:58.985855 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.985862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.986152 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.986719 (+ 567us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:58.986978 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.988201 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.990162 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.991607 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.992804 (+ 1197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.992990 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.993039 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.993153 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9147,"thread_start_us":9358,"threads_started":1}
I20260927 22:51:58.995147 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.985930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35461 (local address 127.0.0.1:59004)
0927 22:51:58.986186 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:58.986205 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:58.986316 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:58.987634 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:58.987638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:58.987660 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:58.987961 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.987970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.989696 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.989700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.991658 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:58.991671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.994399 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.994409 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:58.994585 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.994619 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:51:58.994723 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:51:58.994846 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 22:51:58.995846 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:58.986143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.0.97.2:35461)
0927 22:51:58.986819 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:58.986914 (+ 95us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:58.986941 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:51:58.987044 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:58.987053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:58.987269 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:58.987451 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:58.988108 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.989537 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.991830 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:58.993111 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:58.993234 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:58.993421 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:58.995413 (+ 1992us) server_negotiation.cc:299] Negotiation successful
0927 22:51:58.995577 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":435,"threads_started":1}
I20260927 22:51:58.996366 7441 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9bf08a489048c485292b4fb8268216 (DEFAULT_TABLE table=test-table [id=ca7b47db2c1a44589d9332d8ccb12041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:58.997625 7441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9bf08a489048c485292b4fb8268216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:59.000218 7589 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9bf08a489048c485292b4fb8268216 (DEFAULT_TABLE table=test-table [id=ca7b47db2c1a44589d9332d8ccb12041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:51:59.001406 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9bf08a489048c485292b4fb8268216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:51:59.004921 7828 tablet_bootstrap.cc:492] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: Bootstrap starting.
I20260927 22:51:59.007848 7828 tablet_bootstrap.cc:654] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:59.009541 7828 log.cc:824] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:59.012111 7830 tablet_bootstrap.cc:492] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: Bootstrap starting.
I20260927 22:51:59.013041 7828 tablet_bootstrap.cc:492] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: No bootstrap required, opened a new log
I20260927 22:51:59.013366 7828 ts_tablet_manager.cc:1397] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:51:59.015588 7831 tablet_bootstrap.cc:492] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: Bootstrap starting.
I20260927 22:51:59.016453 7830 tablet_bootstrap.cc:654] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:59.017738 7830 log.cc:824] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:59.018766 7831 tablet_bootstrap.cc:654] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: Neither blocks nor log segments found. Creating new log.
I20260927 22:51:59.019642 7831 log.cc:824] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:51:59.020598 7830 tablet_bootstrap.cc:492] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: No bootstrap required, opened a new log
I20260927 22:51:59.020860 7830 ts_tablet_manager.cc:1397] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:51:59.022014 7831 tablet_bootstrap.cc:492] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: No bootstrap required, opened a new log
I20260927 22:51:59.022246 7831 ts_tablet_manager.cc:1397] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 22:51:59.024183 7828 raft_consensus.cc:348] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } } peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } }
I20260927 22:51:59.024570 7828 raft_consensus.cc:374] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:59.024677 7828 raft_consensus.cc:729] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f009fcb18843b9bebd0475077d28f6, State: Initialized, Role: FOLLOWER
I20260927 22:51:59.025245 7828 consensus_queue.cc:260] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } } peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } }
I20260927 22:51:59.026872 7828 ts_tablet_manager.cc:1428] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260927 22:51:59.027144 7796 heartbeater.cc:503] Master 127.0.97.62:35383 was elected leader, sending a full tablet report...
I20260927 22:51:59.031230 7831 raft_consensus.cc:348] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } } peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } }
I20260927 22:51:59.031625 7831 raft_consensus.cc:374] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:59.031734 7831 raft_consensus.cc:729] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400068953a8d411086ce260244bbdf2d, State: Initialized, Role: FOLLOWER
I20260927 22:51:59.032661 7831 consensus_queue.cc:260] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } } peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } }
I20260927 22:51:59.033802 7830 raft_consensus.cc:348] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } } peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } }
I20260927 22:51:59.034157 7830 raft_consensus.cc:374] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:51:59.034263 7830 raft_consensus.cc:729] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820c3e17809a41d2847c3da4760d7aa0, State: Initialized, Role: FOLLOWER
I20260927 22:51:59.034794 7830 consensus_queue.cc:260] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } } peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } }
I20260927 22:51:59.036434 7830 ts_tablet_manager.cc:1428] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260927 22:51:59.039523 7831 ts_tablet_manager.cc:1428] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20260927 22:51:59.039726 7655 heartbeater.cc:503] Master 127.0.97.62:35383 was elected leader, sending a full tablet report...
I20260927 22:51:59.059252 7834 raft_consensus.cc:482] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:51:59.059512 7834 raft_consensus.cc:504] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } } peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } }
I20260927 22:51:59.060966 7834 leader_election.cc:288] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033), 400068953a8d411086ce260244bbdf2d (127.0.97.2:35461)
I20260927 22:51:59.066668 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.061268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59635 (local address 127.0.97.1:44033)
0927 22:51:59.061463 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.061469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.061486 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:51:59.061667 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.061671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.061747 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.061838 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:59.062190 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.062908 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.064933 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.065753 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.065880 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.065972 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.066308 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.066425 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:51:59.066684 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.061162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44033 (local address 127.0.97.3:59635)
0927 22:51:59.061338 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:59.061359 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:59.061474 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:59.061857 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:59.061861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:59.061886 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:59.062052 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.062060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.063029 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.063037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.064776 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.064787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.065917 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.065922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.066118 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.066174 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:51:59.066285 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:51:59.066411 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":18}
I20260927 22:51:59.066684 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.061620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35947 (local address 127.0.97.2:35461)
0927 22:51:59.061808 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.061816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.061835 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:51:59.061891 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.061902 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.061947 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.062038 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:51:59.062375 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.063043 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.064975 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.065791 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.065922 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.066010 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.066342 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.066459 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:51:59.066715 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.061490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35461 (local address 127.0.97.3:35947)
0927 22:51:59.061649 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 22:51:59.061682 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 22:51:59.061777 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:51:59.062060 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:51:59.062063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:51:59.062088 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:51:59.062243 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.062250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.063196 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.063201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.064819 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:51:59.064830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.065943 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.065949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:51:59.066192 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.066214 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:51:59.066319 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:51:59.066448 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":21}
I20260927 22:51:59.067592 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9bf08a489048c485292b4fb8268216" candidate_uuid: "53f009fcb18843b9bebd0475077d28f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820c3e17809a41d2847c3da4760d7aa0" is_pre_election: true
I20260927 22:51:59.067584 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9bf08a489048c485292b4fb8268216" candidate_uuid: "53f009fcb18843b9bebd0475077d28f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "400068953a8d411086ce260244bbdf2d" is_pre_election: true
I20260927 22:51:59.068002 7461 raft_consensus.cc:2454] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53f009fcb18843b9bebd0475077d28f6 in term 0.
I20260927 22:51:59.068020 7609 raft_consensus.cc:2454] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53f009fcb18843b9bebd0475077d28f6 in term 0.
I20260927 22:51:59.068590 7685 leader_election.cc:302] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f009fcb18843b9bebd0475077d28f6, 820c3e17809a41d2847c3da4760d7aa0; no voters:
I20260927 22:51:59.068938 7834 raft_consensus.cc:2781] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:51:59.069094 7834 raft_consensus.cc:482] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:51:59.069233 7834 raft_consensus.cc:3037] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:59.071403 7834 raft_consensus.cc:504] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } } peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } }
I20260927 22:51:59.071897 7834 leader_election.cc:288] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [CANDIDATE]: Term 1 election: Requested vote from peers 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033), 400068953a8d411086ce260244bbdf2d (127.0.97.2:35461)
I20260927 22:51:59.072304 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9bf08a489048c485292b4fb8268216" candidate_uuid: "53f009fcb18843b9bebd0475077d28f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820c3e17809a41d2847c3da4760d7aa0"
I20260927 22:51:59.072371 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9bf08a489048c485292b4fb8268216" candidate_uuid: "53f009fcb18843b9bebd0475077d28f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "400068953a8d411086ce260244bbdf2d"
I20260927 22:51:59.072461 7461 raft_consensus.cc:3037] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:59.072587 7609 raft_consensus.cc:3037] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:51:59.074631 7461 raft_consensus.cc:2454] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f009fcb18843b9bebd0475077d28f6 in term 1.
I20260927 22:51:59.074895 7609 raft_consensus.cc:2454] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f009fcb18843b9bebd0475077d28f6 in term 1.
I20260927 22:51:59.075014 7685 leader_election.cc:302] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f009fcb18843b9bebd0475077d28f6, 820c3e17809a41d2847c3da4760d7aa0; no voters:
I20260927 22:51:59.075290 7834 raft_consensus.cc:2781] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 1 FOLLOWER]: Leader election won for term 1
W20260927 22:51:59.075717 7663 tablet.cc:2369] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:51:59.075963 7834 raft_consensus.cc:686] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 1 LEADER]: Becoming Leader. State: Replica: 53f009fcb18843b9bebd0475077d28f6, State: Running, Role: LEADER
I20260927 22:51:59.076628 7834 consensus_queue.cc:237] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } } peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } }
I20260927 22:51:59.081321 6541 catalog_manager.cc:5495] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53f009fcb18843b9bebd0475077d28f6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "53f009fcb18843b9bebd0475077d28f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } health_report { overall_health: UNKNOWN } } }
I20260927 22:51:59.112437 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.107581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.0.97.3:35043)
0927 22:51:59.107742 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.107746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.107761 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:51:59.107913 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.107918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.108008 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.108122 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:51:59.108397 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.109159 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.110373 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.110841 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.110947 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.111345 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:51:59.111888 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:51:59.112025 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.112111 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.112166 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:51:59.119999 7461 raft_consensus.cc:1261] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0 [term 1 FOLLOWER]: Refusing update from remote peer 53f009fcb18843b9bebd0475077d28f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:59.120038 7609 raft_consensus.cc:1261] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d [term 1 FOLLOWER]: Refusing update from remote peer 53f009fcb18843b9bebd0475077d28f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:51:59.121249 7837 consensus_queue.cc:1035] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 22:51:59.121567 7834 consensus_queue.cc:1035] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:51:59.132761 7840 mvcc.cc:205] Tried to move back new op lower bound from 7334090830306299904 to 7334090830142038016. Current Snapshot: MvccSnapshot[applied={T|T < 7334090830306299904}]
I20260927 22:51:59.133625 7841 mvcc.cc:205] Tried to move back new op lower bound from 7334090830306299904 to 7334090830142038016. Current Snapshot: MvccSnapshot[applied={T|T < 7334090830306299904}]
I20260927 22:51:59.158824 6541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:59.159062 6541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:51:59.164189 6541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6be24da53c1245a8afab32e4c229cc0f: Sending DeleteTablet for 3 replicas of tablet 1d9bf08a489048c485292b4fb8268216
I20260927 22:51:59.165172 7730 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d9bf08a489048c485292b4fb8268216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46720
I20260927 22:51:59.167294 7441 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d9bf08a489048c485292b4fb8268216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40346
I20260927 22:51:59.168030 7589 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d9bf08a489048c485292b4fb8268216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:51:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59004
I20260927 22:51:59.179772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 6509
I20260927 22:51:59.180614 6571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:51:59.192134 7853 tablet_replica.cc:330] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: stopping tablet replica
I20260927 22:51:59.192906 7852 tablet_replica.cc:330] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: stopping tablet replica
I20260927 22:51:59.193571 7853 raft_consensus.cc:2229] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:59.194662 7853 raft_consensus.cc:2258] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:59.194931 7852 raft_consensus.cc:2229] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:51:59.195513 7852 raft_consensus.cc:2258] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:59.198797 7852 ts_tablet_manager.cc:1905] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:59.199950 7853 ts_tablet_manager.cc:1905] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:59.206707 7853 ts_tablet_manager.cc:1918] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:59.211035 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.167432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.0.97.61:38543)
0927 22:51:59.167583 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.167589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.178784 (+ 11195us) server_negotiation.cc:407] Connection header received
0927 22:51:59.189691 (+ 10907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.189696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.189792 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.189864 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:59.194308 (+ 4444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.195412 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.198339 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.199004 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.199101 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.202959 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:59.202966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:59.202971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:59.203009 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:59.206463 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.207246 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.207253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.207257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.207386 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.207850 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.207885 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.207890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.208551 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:59.208828 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.210302 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.210522 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 22:51:59.211954 7850 tablet_replica.cc:330] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: stopping tablet replica
I20260927 22:51:59.212513 7850 raft_consensus.cc:2229] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:51:59.213744 7853 log.cc:1196] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/1d9bf08a489048c485292b4fb8268216
I20260927 22:51:59.214565 7850 raft_consensus.cc:2258] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:51:59.216926 7850 ts_tablet_manager.cc:1905] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:51:59.219146 7853 ts_tablet_manager.cc:1939] T 1d9bf08a489048c485292b4fb8268216 P 400068953a8d411086ce260244bbdf2d: Deleting consensus metadata
I20260927 22:51:59.221352 7852 ts_tablet_manager.cc:1918] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:59.221532 7852 log.cc:1196] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/1d9bf08a489048c485292b4fb8268216
I20260927 22:51:59.222164 7852 ts_tablet_manager.cc:1939] T 1d9bf08a489048c485292b4fb8268216 P 820c3e17809a41d2847c3da4760d7aa0: Deleting consensus metadata
I20260927 22:51:59.230057 7850 ts_tablet_manager.cc:1918] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:51:59.230247 7850 log.cc:1196] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/1d9bf08a489048c485292b4fb8268216
I20260927 22:51:59.230789 7850 ts_tablet_manager.cc:1939] T 1d9bf08a489048c485292b4fb8268216 P 53f009fcb18843b9bebd0475077d28f6: Deleting consensus metadata
I20260927 22:51:59.231858 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.179239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.0.97.60:46589)
0927 22:51:59.179434 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.179443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.213274 (+ 33831us) server_negotiation.cc:407] Connection header received
0927 22:51:59.213431 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.213436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.213557 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.213701 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:59.215887 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.216977 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.220830 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.221502 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.221649 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.225014 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:59.225021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:59.225027 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:59.225076 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:59.228688 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.229517 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.229523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.229526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.229695 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.230152 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.230158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.230161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.230475 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:59.230692 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.231254 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.231456 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:51:59.329069 6528 catalog_manager.cc:4841] TS 53f009fcb18843b9bebd0475077d28f6 (127.0.97.3:35043): tablet 1d9bf08a489048c485292b4fb8268216 (table test-table [id=ca7b47db2c1a44589d9332d8ccb12041]) successfully deleted
I20260927 22:51:59.330046 6527 catalog_manager.cc:4841] TS 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033): tablet 1d9bf08a489048c485292b4fb8268216 (table test-table [id=ca7b47db2c1a44589d9332d8ccb12041]) successfully deleted
I20260927 22:51:59.330291 6527 catalog_manager.cc:4841] TS 400068953a8d411086ce260244bbdf2d (127.0.97.2:35461): tablet 1d9bf08a489048c485292b4fb8268216 (table test-table [id=ca7b47db2c1a44589d9332d8ccb12041]) successfully deleted
I20260927 22:51:59.330790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 6509
I20260927 22:51:59.384613 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.356539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.0.97.61:38543)
0927 22:51:59.356725 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.356731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.367438 (+ 10707us) server_negotiation.cc:407] Connection header received
0927 22:51:59.367921 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.367927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.368011 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.368082 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:59.370492 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.371661 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.374380 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.375026 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.375117 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.378642 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:59.378647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:59.378650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:59.378681 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:59.381741 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.382353 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.382362 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.382367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.382520 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.382987 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.382993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.382997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.383402 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:59.383659 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.384023 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.384251 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:51:59.389941 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:51:59.356055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.0.97.60:46589)
0927 22:51:59.356264 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:51:59.356273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:51:59.374224 (+ 17951us) server_negotiation.cc:407] Connection header received
0927 22:51:59.375481 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:51:59.375488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:51:59.375573 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:51:59.375686 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:51:59.378014 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.379099 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.380347 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:51:59.380986 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:51:59.381111 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:51:59.384498 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:51:59.384505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:51:59.384511 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:51:59.384550 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:51:59.387446 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.388063 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.388069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.388072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.388198 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:51:59.388558 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:51:59.388563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:51:59.388568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:51:59.388830 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:51:59.389096 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:51:59.389400 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 22:51:59.389638 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:52:00.151273 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.150373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.1:37685)
0927 22:52:00.150945 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.151078 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"threads_started":1}
W20260927 22:52:00.151980 7507 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35383 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:00.153045 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.152279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:49333)
0927 22:52:00.152661 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.152841 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":195,"threads_started":1}
W20260927 22:52:00.153398 7655 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35383 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:00.155803 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.154978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.3:37755)
0927 22:52:00.155409 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.155585 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":243,"threads_started":1}
W20260927 22:52:00.156107 7796 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35383 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:00.664788 7865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6be24da53c1245a8afab32e4c229cc0f)
I20260927 22:52:00.665108 7866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6be24da53c1245a8afab32e4c229cc0f)
I20260927 22:52:00.665091 7865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } }
I20260927 22:52:00.665354 7866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } }
I20260927 22:52:00.666545 7865 leader_election.cc:288] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383), 2a73ae77b7f54c929293c60d37f40189 (127.0.97.61:38543)
I20260927 22:52:00.672696 7866 leader_election.cc:288] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383), 2a563337cda04b02bde7e5909c712343 (127.0.97.60:46589)
I20260927 22:52:00.679481 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.666598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45714)
0927 22:52:00.678891 (+ 12293us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.679110 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12119,"threads_started":1}
W20260927 22:52:00.680109 7310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:00.682651 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.668645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45728)
0927 22:52:00.682092 (+ 13447us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.682301 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13346,"thread_start_us":8139,"threads_started":1}
W20260927 22:52:00.683001 7122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:00.685388 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.684999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45730)
0927 22:52:00.685148 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.685231 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:52:00.685778 7310 leader_election.cc:334] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:00.685977 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.685569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45742)
0927 22:52:00.685743 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.685828 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 22:52:00.686161 7122 leader_election.cc:334] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:00.691272 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.666846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46589 (local address 127.0.0.1:51432)
0927 22:52:00.677480 (+ 10634us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.677508 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:00.677698 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:00.683869 (+ 6171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:00.683878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:00.683917 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:00.684345 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:00.684364 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.686490 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.686494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:00.688566 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:00.688583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.690351 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.690358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:00.690561 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:00.690595 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:52:00.690713 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:52:00.690847 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10096,"mutex_wait_us":6,"thread_start_us":15018,"threads_started":1}
I20260927 22:52:00.691273 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.666861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.0.1:52926)
0927 22:52:00.681357 (+ 14496us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.681382 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:00.681533 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:00.684221 (+ 2688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:00.684228 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:00.684275 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:00.685001 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:00.685019 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.686741 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.686746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:00.688664 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:00.688680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.690351 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.690358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:00.690597 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:00.690628 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:52:00.690713 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:52:00.690842 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14083,"mutex_wait_us":283,"thread_start_us":14190,"threads_started":1}
I20260927 22:52:00.691349 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.667119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.0.97.60:46589)
0927 22:52:00.683346 (+ 16227us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:00.683355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:00.683378 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:52:00.683463 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:00.683472 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:00.683573 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:00.684529 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:00.684849 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.686345 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.688773 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.690178 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.690360 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:00.690507 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:00.690749 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 22:52:00.690893 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":16059,"thread_start_us":15892,"threads_started":1}
I20260927 22:52:00.691332 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.682124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.0.97.61:38543)
0927 22:52:00.683602 (+ 1478us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:00.683611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:00.683636 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:52:00.683721 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:00.683729 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:00.683823 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:00.683965 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:00.685243 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.686596 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.688832 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:00.690175 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:00.690319 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:00.690505 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:00.690749 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 22:52:00.690903 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":890,"server-negotiator.queue_time_us":1318,"thread_start_us":1557,"threads_started":1}
I20260927 22:52:00.692196 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a563337cda04b02bde7e5909c712343" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2a73ae77b7f54c929293c60d37f40189" is_pre_election: true
I20260927 22:52:00.692201 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a73ae77b7f54c929293c60d37f40189" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2a563337cda04b02bde7e5909c712343" is_pre_election: true
I20260927 22:52:00.692505 7148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a563337cda04b02bde7e5909c712343 in term 1.
I20260927 22:52:00.692538 7337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a73ae77b7f54c929293c60d37f40189 in term 1.
I20260927 22:52:00.692996 7125 leader_election.cc:302] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a563337cda04b02bde7e5909c712343, 2a73ae77b7f54c929293c60d37f40189; no voters: 6be24da53c1245a8afab32e4c229cc0f
I20260927 22:52:00.693010 7312 leader_election.cc:302] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a563337cda04b02bde7e5909c712343, 2a73ae77b7f54c929293c60d37f40189; no voters: 6be24da53c1245a8afab32e4c229cc0f
I20260927 22:52:00.693336 7865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 22:52:00.693465 7865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6be24da53c1245a8afab32e4c229cc0f)
I20260927 22:52:00.693565 7865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:52:00.693578 7866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 22:52:00.693710 7866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6be24da53c1245a8afab32e4c229cc0f)
I20260927 22:52:00.693830 7866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:52:00.695675 7865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } }
I20260927 22:52:00.696053 7866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } }
I20260927 22:52:00.696287 7865 leader_election.cc:288] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 2 election: Requested vote from peers 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383), 2a73ae77b7f54c929293c60d37f40189 (127.0.97.61:38543)
I20260927 22:52:00.696801 7866 leader_election.cc:288] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [CANDIDATE]: Term 2 election: Requested vote from peers 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383), 2a563337cda04b02bde7e5909c712343 (127.0.97.60:46589)
I20260927 22:52:00.696856 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.696378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45744)
0927 22:52:00.696647 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.696728 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 22:52:00.696837 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a563337cda04b02bde7e5909c712343" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2a73ae77b7f54c929293c60d37f40189"
I20260927 22:52:00.697146 7148 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a563337cda04b02bde7e5909c712343 in current term 2: Already voted for candidate 2a73ae77b7f54c929293c60d37f40189 in this term.
I20260927 22:52:00.697300 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.696840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45754)
0927 22:52:00.697047 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.697153 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 22:52:00.697438 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a73ae77b7f54c929293c60d37f40189" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2a563337cda04b02bde7e5909c712343"
I20260927 22:52:00.697770 7337 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a73ae77b7f54c929293c60d37f40189 in current term 2: Already voted for candidate 2a563337cda04b02bde7e5909c712343 in this term.
I20260927 22:52:00.697785 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.697404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45762)
0927 22:52:00.697528 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.697632 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 22:52:00.698035 7310 leader_election.cc:334] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:00.698273 7312 leader_election.cc:302] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a563337cda04b02bde7e5909c712343; no voters: 2a73ae77b7f54c929293c60d37f40189, 6be24da53c1245a8afab32e4c229cc0f
I20260927 22:52:00.698637 7865 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 22:52:00.698769 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:00.698392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:45770)
0927 22:52:00.698522 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:00.698614 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 22:52:00.698930 7122 leader_election.cc:334] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:00.699033 7122 leader_election.cc:302] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a73ae77b7f54c929293c60d37f40189; no voters: 2a563337cda04b02bde7e5909c712343, 6be24da53c1245a8afab32e4c229cc0f
I20260927 22:52:00.699282 7866 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 22:52:01.153646 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:01.152649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.1:43201)
0927 22:52:01.153226 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:01.153364 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":339,"threads_started":1}
I20260927 22:52:01.154757 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:01.154034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:50229)
0927 22:52:01.154522 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:01.154602 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":273,"threads_started":1}
I20260927 22:52:01.157485 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:01.156689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.3:60159)
0927 22:52:01.157171 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:01.157283 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":327,"threads_started":1}
I20260927 22:52:02.155462 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.154529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.1:35441)
0927 22:52:02.155139 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.155239 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":360,"threads_started":1}
I20260927 22:52:02.156217 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.155484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:40975)
0927 22:52:02.155913 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.156043 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":248,"threads_started":1}
I20260927 22:52:02.159288 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.158257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.3:59701)
0927 22:52:02.159056 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.159146 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":622,"threads_started":1}
I20260927 22:52:02.478276 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:52:02.478502 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } }
I20260927 22:52:02.479100 7881 leader_election.cc:288] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383), 2a73ae77b7f54c929293c60d37f40189 (127.0.97.61:38543)
I20260927 22:52:02.479606 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a563337cda04b02bde7e5909c712343" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2a73ae77b7f54c929293c60d37f40189" is_pre_election: true
I20260927 22:52:02.479864 7148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a563337cda04b02bde7e5909c712343 in term 2.
I20260927 22:52:02.479998 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.479189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:35932)
0927 22:52:02.479629 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.479738 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":214,"threads_started":1}
I20260927 22:52:02.480286 7312 leader_election.cc:302] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a563337cda04b02bde7e5909c712343, 2a73ae77b7f54c929293c60d37f40189; no voters:
I20260927 22:52:02.480651 7881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 22:52:02.480805 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:52:02.480906 7881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:52:02.481562 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.481197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:35936)
0927 22:52:02.481361 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.481440 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 22:52:02.481743 7310 leader_election.cc:334] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:02.483191 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } }
I20260927 22:52:02.483683 7881 leader_election.cc:288] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 3 election: Requested vote from peers 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383), 2a73ae77b7f54c929293c60d37f40189 (127.0.97.61:38543)
I20260927 22:52:02.484148 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.483795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:35944)
0927 22:52:02.483913 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.484006 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:52:02.484119 7148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a563337cda04b02bde7e5909c712343" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2a73ae77b7f54c929293c60d37f40189"
I20260927 22:52:02.484308 7148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:52:02.485009 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.484668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:35952)
0927 22:52:02.484805 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.484887 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:52:02.485183 7310 leader_election.cc:334] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383): Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
I20260927 22:52:02.486606 7148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a563337cda04b02bde7e5909c712343 in term 3.
I20260927 22:52:02.486939 7312 leader_election.cc:302] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a563337cda04b02bde7e5909c712343, 2a73ae77b7f54c929293c60d37f40189; no voters: 6be24da53c1245a8afab32e4c229cc0f
I20260927 22:52:02.487174 7881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:52:02.487574 7881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [term 3 LEADER]: Becoming Leader. State: Replica: 2a563337cda04b02bde7e5909c712343, State: Running, Role: LEADER
I20260927 22:52:02.487915 7881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } }
I20260927 22:52:02.489732 7884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a563337cda04b02bde7e5909c712343. Latest consensus state: current_term: 3 leader_uuid: "2a563337cda04b02bde7e5909c712343" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:52:02.490110 7884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:02.490726 7886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:02.494069 7886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:02.494824 7886 catalog_manager.cc:1259] Loaded cluster ID: a5c4f88810be476fa78002bf583faa9c
I20260927 22:52:02.494892 7886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:02.495978 7886 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:02.496701 7886 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Loaded TSK: 0
I20260927 22:52:02.497470 7886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:02.562255 7326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:52:02.568925 7326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:52:02.573118 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.572668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:35980)
0927 22:52:02.572870 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.572966 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 22:52:02.573144 7148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [term 3 FOLLOWER]: Refusing update from remote peer 2a563337cda04b02bde7e5909c712343: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 22:52:02.573846 7884 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 22:52:02.574167 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.573786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:35984)
0927 22:52:02.573963 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.574024 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:52:02.574324 7310 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 -> Peer 6be24da53c1245a8afab32e4c229cc0f (127.0.97.62:35383): Couldn't send request to peer 6be24da53c1245a8afab32e4c229cc0f. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:52:02.576404 7888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a563337cda04b02bde7e5909c712343. Latest consensus state: current_term: 3 leader_uuid: "2a563337cda04b02bde7e5909c712343" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:52:02.576704 7888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:02.578076 7884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2a563337cda04b02bde7e5909c712343" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:52:02.578336 7884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:02.578962 7888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2a563337cda04b02bde7e5909c712343" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6be24da53c1245a8afab32e4c229cc0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35383 } } peers { permanent_uuid: "2a73ae77b7f54c929293c60d37f40189" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38543 } attrs { promote: false } } peers { permanent_uuid: "2a563337cda04b02bde7e5909c712343" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46589 } attrs { promote: false } } }
I20260927 22:52:02.579205 7888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a73ae77b7f54c929293c60d37f40189 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:52:02.593994 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.585937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35461 (local address 127.0.0.1:53748)
0927 22:52:02.586073 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.586093 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.586214 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.587082 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.587086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.587112 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:02.587428 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.587441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.589323 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.589332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.591617 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.591630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.593159 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.593165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.593351 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.593375 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.593480 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.593654 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":82}
I20260927 22:52:02.593977 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.586244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.0.97.2:35461)
0927 22:52:02.586722 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.586730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.586752 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:52:02.586825 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.586829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.586914 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.587586 (+ 672us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:02.587798 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.589124 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.591974 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.592954 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.593102 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.593201 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.593504 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.593668 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":263,"threads_started":1}
I20260927 22:52:02.594846 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.586324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35043 (local address 127.0.0.1:49520)
0927 22:52:02.586854 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.586875 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.586998 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.587329 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.587333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.587359 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:02.587678 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.587686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.589501 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.589509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.591855 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.591869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.593985 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.594000 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.594222 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.594264 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.594350 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.594482 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":72,"thread_start_us":435,"threads_started":1}
I20260927 22:52:02.594872 7589 tablet_service.cc:1467] Processing CreateTablet for tablet 1a000bfe5927463eb8d5f761c792d92c (DEFAULT_TABLE table=test-table [id=fb1a75a78ab941cfa3a86f552c6d67ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:02.595302 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a000bfe5927463eb8d5f761c792d92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:02.595460 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.586615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.0.97.3:35043)
0927 22:52:02.587071 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.587077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.587094 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:52:02.587153 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.587165 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.587215 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.587316 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:02.587838 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.589359 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.592033 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.593652 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.593768 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.593871 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.595039 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.595173 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":260,"threads_started":1}
I20260927 22:52:02.596623 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.587698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44033 (local address 127.0.0.1:46182)
0927 22:52:02.588204 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.588221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.588324 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.590334 (+ 2010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.590339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.590364 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:02.590816 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.590826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.592171 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.592175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.594279 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.594291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.595814 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.595820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.596024 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.596045 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.596157 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.596271 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":19,"thread_start_us":464,"threads_started":1}
I20260927 22:52:02.596635 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.588044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.0.97.1:44033)
0927 22:52:02.589999 (+ 1955us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.590006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.590024 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:52:02.590089 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.590094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.590171 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.590285 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:02.591002 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.592031 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.594445 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.595629 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.595775 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.595872 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.596183 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.596306 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1825,"thread_start_us":1709,"threads_started":1}
I20260927 22:52:02.598294 7730 tablet_service.cc:1467] Processing CreateTablet for tablet 1a000bfe5927463eb8d5f761c792d92c (DEFAULT_TABLE table=test-table [id=fb1a75a78ab941cfa3a86f552c6d67ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:02.598729 7730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a000bfe5927463eb8d5f761c792d92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:02.598717 7441 tablet_service.cc:1467] Processing CreateTablet for tablet 1a000bfe5927463eb8d5f761c792d92c (DEFAULT_TABLE table=test-table [id=fb1a75a78ab941cfa3a86f552c6d67ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:02.599117 7441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a000bfe5927463eb8d5f761c792d92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:02.601079 7899 tablet_bootstrap.cc:492] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d: Bootstrap starting.
I20260927 22:52:02.602936 7899 tablet_bootstrap.cc:654] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:02.605372 7899 tablet_bootstrap.cc:492] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d: No bootstrap required, opened a new log
I20260927 22:52:02.605542 7899 ts_tablet_manager.cc:1397] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 22:52:02.606559 7899 raft_consensus.cc:348] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } } peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } }
I20260927 22:52:02.606860 7899 raft_consensus.cc:374] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:02.606972 7899 raft_consensus.cc:729] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400068953a8d411086ce260244bbdf2d, State: Initialized, Role: FOLLOWER
I20260927 22:52:02.607259 7899 consensus_queue.cc:260] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } } peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } }
I20260927 22:52:02.608496 7899 ts_tablet_manager.cc:1428] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:52:02.609119 7902 tablet_bootstrap.cc:492] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: Bootstrap starting.
I20260927 22:52:02.610146 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.609548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:33341)
0927 22:52:02.609890 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.609988 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260927 22:52:02.610944 7902 tablet_bootstrap.cc:654] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:02.612615 7654 heartbeater.cc:503] Master 127.0.97.60:46589 was elected leader, sending a full tablet report...
I20260927 22:52:02.613770 7901 tablet_bootstrap.cc:492] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: Bootstrap starting.
I20260927 22:52:02.614178 7902 tablet_bootstrap.cc:492] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: No bootstrap required, opened a new log
I20260927 22:52:02.614368 7902 ts_tablet_manager.cc:1397] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260927 22:52:02.615190 7902 raft_consensus.cc:348] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } } peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } }
I20260927 22:52:02.615489 7902 raft_consensus.cc:374] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:02.615603 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.615115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:49707)
0927 22:52:02.615343 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.615440 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 22:52:02.615621 7902 raft_consensus.cc:729] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820c3e17809a41d2847c3da4760d7aa0, State: Initialized, Role: FOLLOWER
I20260927 22:52:02.615661 7901 tablet_bootstrap.cc:654] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:02.615885 7902 consensus_queue.cc:260] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } } peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } }
I20260927 22:52:02.616828 7902 ts_tablet_manager.cc:1428] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:52:02.617965 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.617348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.1:32949)
0927 22:52:02.617706 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.617798 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
I20260927 22:52:02.619153 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.618782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.1:59555)
0927 22:52:02.618917 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.618987 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 22:52:02.619210 7506 heartbeater.cc:503] Master 127.0.97.60:46589 was elected leader, sending a full tablet report...
I20260927 22:52:02.622717 7901 tablet_bootstrap.cc:492] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: No bootstrap required, opened a new log
I20260927 22:52:02.622855 7901 ts_tablet_manager.cc:1397] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 22:52:02.623457 7901 raft_consensus.cc:348] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } } peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } }
I20260927 22:52:02.623627 7901 raft_consensus.cc:374] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:02.623736 7901 raft_consensus.cc:729] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f009fcb18843b9bebd0475077d28f6, State: Initialized, Role: FOLLOWER
I20260927 22:52:02.623950 7901 consensus_queue.cc:260] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } } peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } }
I20260927 22:52:02.624660 7901 ts_tablet_manager.cc:1428] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:52:02.625370 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.624940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.3:43711)
0927 22:52:02.625130 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.625215 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 22:52:02.626423 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.626016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.3:33397)
0927 22:52:02.626188 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.626263 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:52:02.626579 7795 heartbeater.cc:503] Master 127.0.97.60:46589 was elected leader, sending a full tablet report...
I20260927 22:52:02.641695 7903 raft_consensus.cc:482] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:52:02.641897 7903 raft_consensus.cc:504] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } } peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } }
I20260927 22:52:02.643306 7903 leader_election.cc:288] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53f009fcb18843b9bebd0475077d28f6 (127.0.97.3:35043), 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033)
W20260927 22:52:02.647356 7798 tablet.cc:2369] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:52:02.650607 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.643625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37725 (local address 127.0.97.1:44033)
0927 22:52:02.643836 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.643842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.644372 (+ 530us) server_negotiation.cc:407] Connection header received
0927 22:52:02.644553 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.644557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.644608 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.644677 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:02.645142 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.646061 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.648314 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.649504 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.649679 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.649769 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.650247 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.650370 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:52:02.650630 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.643585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42853 (local address 127.0.97.3:35043)
0927 22:52:02.643788 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.643797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.643822 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:52:02.644009 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.644015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.644081 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.644182 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:02.644758 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.645752 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.648308 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.649502 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.649677 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.649769 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.650247 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.650370 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:52:02.650718 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.643459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35043 (local address 127.0.97.2:42853)
0927 22:52:02.643704 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.643723 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.643838 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.644217 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.644222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.644249 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:02.644611 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.644620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.645876 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.645880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.648139 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.648162 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.649746 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.649750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.650003 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.650117 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.650211 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.650359 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":81}
I20260927 22:52:02.650786 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.643519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44033 (local address 127.0.97.2:37725)
0927 22:52:02.644267 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.644282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:02.644429 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:02.644718 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:02.644722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:02.644749 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:02.645009 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.645017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.646173 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.646178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.648161 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:02.648170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.649738 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.649744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:02.650018 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.650120 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 22:52:02.650212 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:52:02.650359 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":139,"spinlock_wait_cycles":11520,"thread_start_us":454,"threads_started":1}
I20260927 22:52:02.651325 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a000bfe5927463eb8d5f761c792d92c" candidate_uuid: "400068953a8d411086ce260244bbdf2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820c3e17809a41d2847c3da4760d7aa0" is_pre_election: true
I20260927 22:52:02.651605 7461 raft_consensus.cc:2454] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 400068953a8d411086ce260244bbdf2d in term 0.
I20260927 22:52:02.651623 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a000bfe5927463eb8d5f761c792d92c" candidate_uuid: "400068953a8d411086ce260244bbdf2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f009fcb18843b9bebd0475077d28f6" is_pre_election: true
I20260927 22:52:02.652005 7750 raft_consensus.cc:2454] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 400068953a8d411086ce260244bbdf2d in term 0.
I20260927 22:52:02.652097 7542 leader_election.cc:302] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 400068953a8d411086ce260244bbdf2d, 820c3e17809a41d2847c3da4760d7aa0; no voters:
I20260927 22:52:02.652521 7903 raft_consensus.cc:2781] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:52:02.652655 7903 raft_consensus.cc:482] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:52:02.652782 7903 raft_consensus.cc:3037] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:02.655067 7903 raft_consensus.cc:504] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } } peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } }
I20260927 22:52:02.655616 7903 leader_election.cc:288] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [CANDIDATE]: Term 1 election: Requested vote from peers 53f009fcb18843b9bebd0475077d28f6 (127.0.97.3:35043), 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033)
I20260927 22:52:02.656066 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a000bfe5927463eb8d5f761c792d92c" candidate_uuid: "400068953a8d411086ce260244bbdf2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f009fcb18843b9bebd0475077d28f6"
I20260927 22:52:02.656075 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a000bfe5927463eb8d5f761c792d92c" candidate_uuid: "400068953a8d411086ce260244bbdf2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820c3e17809a41d2847c3da4760d7aa0"
I20260927 22:52:02.656270 7461 raft_consensus.cc:3037] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:02.656270 7750 raft_consensus.cc:3037] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:02.659204 7461 raft_consensus.cc:2454] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 400068953a8d411086ce260244bbdf2d in term 1.
I20260927 22:52:02.659205 7750 raft_consensus.cc:2454] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 400068953a8d411086ce260244bbdf2d in term 1.
I20260927 22:52:02.659662 7542 leader_election.cc:302] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 400068953a8d411086ce260244bbdf2d, 820c3e17809a41d2847c3da4760d7aa0; no voters:
I20260927 22:52:02.659935 7903 raft_consensus.cc:2781] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:02.660377 7903 raft_consensus.cc:686] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 1 LEADER]: Becoming Leader. State: Replica: 400068953a8d411086ce260244bbdf2d, State: Running, Role: LEADER
I20260927 22:52:02.660785 7903 consensus_queue.cc:237] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } } peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } }
I20260927 22:52:02.661326 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.660902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:37857)
0927 22:52:02.661057 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.661190 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 22:52:02.663921 7327 catalog_manager.cc:5495] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 400068953a8d411086ce260244bbdf2d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "400068953a8d411086ce260244bbdf2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "400068953a8d411086ce260244bbdf2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 } health_report { overall_health: UNKNOWN } } }
I20260927 22:52:02.679277 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.674551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.0.97.2:35461)
0927 22:52:02.674700 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.674704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.674720 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:52:02.674862 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.674865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.674941 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.675133 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:02.675403 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.676145 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.677420 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.677911 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.678011 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.678368 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:02.678857 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:02.678976 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.679041 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.679097 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:52:02.682972 7461 raft_consensus.cc:1261] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0 [term 1 FOLLOWER]: Refusing update from remote peer 400068953a8d411086ce260244bbdf2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:02.683207 7750 raft_consensus.cc:1261] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6 [term 1 FOLLOWER]: Refusing update from remote peer 400068953a8d411086ce260244bbdf2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:02.683707 7909 consensus_queue.cc:1035] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "820c3e17809a41d2847c3da4760d7aa0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:02.684109 7903 consensus_queue.cc:1035] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f009fcb18843b9bebd0475077d28f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:02.686633 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.686063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.3:47015)
0927 22:52:02.686299 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.686412 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 22:52:02.688055 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.687507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.1:57217)
0927 22:52:02.687746 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.687862 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:52:02.693389 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.692963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.3:46325)
0927 22:52:02.693144 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.693226 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 22:52:02.694025 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.693655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:60015)
0927 22:52:02.693785 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.693869 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 22:52:02.703178 7326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:02.703415 7326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:02.712935 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.711703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.2:48031)
0927 22:52:02.712586 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.712724 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":721}
I20260927 22:52:02.715857 7326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343: Sending DeleteTablet for 3 replicas of tablet 1a000bfe5927463eb8d5f761c792d92c
W20260927 22:52:02.716590 7509 tablet.cc:2369] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:52:02.717509 7730 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a000bfe5927463eb8d5f761c792d92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:02 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:49520
I20260927 22:52:02.717527 7589 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a000bfe5927463eb8d5f761c792d92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:02 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53748
I20260927 22:52:02.717962 7440 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a000bfe5927463eb8d5f761c792d92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:02 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46182
I20260927 22:52:02.718176 7922 tablet_replica.cc:330] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: stopping tablet replica
I20260927 22:52:02.718219 7441 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a000bfe5927463eb8d5f761c792d92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:02 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46182
I20260927 22:52:02.718349 7922 raft_consensus.cc:2229] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:02.718662 7922 raft_consensus.cc:2258] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:02.719468 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 7314
I20260927 22:52:02.719735 7922 ts_tablet_manager.cc:1905] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:02.727443 7922 ts_tablet_manager.cc:1918] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:02.727742 7922 log.cc:1196] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/1a000bfe5927463eb8d5f761c792d92c
I20260927 22:52:02.728341 7922 ts_tablet_manager.cc:1939] T 1a000bfe5927463eb8d5f761c792d92c P 53f009fcb18843b9bebd0475077d28f6: Deleting consensus metadata
I20260927 22:52:02.729552 7313 catalog_manager.cc:4841] TS 53f009fcb18843b9bebd0475077d28f6 (127.0.97.3:35043): tablet 1a000bfe5927463eb8d5f761c792d92c (table test-table [id=fb1a75a78ab941cfa3a86f552c6d67ed]) successfully deleted
I20260927 22:52:02.733963 7921 tablet_replica.cc:330] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d: stopping tablet replica
I20260927 22:52:02.734234 7921 raft_consensus.cc:2229] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:52:02.734968 7921 raft_consensus.cc:2258] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:02.737035 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.736343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.97.1:58355)
0927 22:52:02.736731 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:02.736819 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276}
I20260927 22:52:02.737151 7921 ts_tablet_manager.cc:1905] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:02.737358 7923 tablet_replica.cc:330] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: stopping tablet replica
I20260927 22:52:02.737628 7923 raft_consensus.cc:2229] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:02.738046 7923 raft_consensus.cc:2258] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:02.738942 7923 ts_tablet_manager.cc:1905] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:02.743772 7921 ts_tablet_manager.cc:1918] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:02.743955 7921 log.cc:1196] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/1a000bfe5927463eb8d5f761c792d92c
I20260927 22:52:02.744503 7921 ts_tablet_manager.cc:1939] T 1a000bfe5927463eb8d5f761c792d92c P 400068953a8d411086ce260244bbdf2d: Deleting consensus metadata
I20260927 22:52:02.745627 7923 ts_tablet_manager.cc:1918] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:02.745627 7312 catalog_manager.cc:4841] TS 400068953a8d411086ce260244bbdf2d (127.0.97.2:35461): tablet 1a000bfe5927463eb8d5f761c792d92c (table test-table [id=fb1a75a78ab941cfa3a86f552c6d67ed]) successfully deleted
I20260927 22:52:02.745837 7923 log.cc:1196] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/1a000bfe5927463eb8d5f761c792d92c
I20260927 22:52:02.746340 7923 ts_tablet_manager.cc:1939] T 1a000bfe5927463eb8d5f761c792d92c P 820c3e17809a41d2847c3da4760d7aa0: Deleting consensus metadata
I20260927 22:52:02.747350 7312 catalog_manager.cc:4841] TS 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033): tablet 1a000bfe5927463eb8d5f761c792d92c (table test-table [id=fb1a75a78ab941cfa3a86f552c6d67ed]) successfully deleted
W20260927 22:52:02.747803 7312 catalog_manager.cc:4808] TS 820c3e17809a41d2847c3da4760d7aa0 (127.0.97.1:44033): delete failed for tablet 1a000bfe5927463eb8d5f761c792d92c because the tablet was not found. No further retry: Not found: Tablet not found: 1a000bfe5927463eb8d5f761c792d92c
I20260927 22:52:02.751286 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.721899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.0.97.1:44033)
0927 22:52:02.722083 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.722092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.734090 (+ 11998us) server_negotiation.cc:407] Connection header received
0927 22:52:02.734329 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.734335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.734413 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.734480 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:02.736520 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.737780 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.739560 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.740240 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.740343 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.744544 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:02.744558 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:02.744578 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:02.744617 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:02.748543 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:02.749171 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:02.749181 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:02.749185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:02.749341 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:02.749765 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:02.749772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:02.749776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:02.750229 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:02.750520 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.750704 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.750908 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:52:02.751894 7501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:02.844699 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 7314
I20260927 22:52:02.868145 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 7518
I20260927 22:52:02.885318 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:02.868884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.0.97.2:35461)
0927 22:52:02.869082 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:02.869094 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:02.869175 (+ 81us) server_negotiation.cc:407] Connection header received
0927 22:52:02.869382 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:02.869394 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:02.869465 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:02.869576 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:02.870911 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.872162 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.873926 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:02.874678 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:02.874822 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:02.878041 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:02.878059 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:02.878079 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:02.878124 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:02.882263 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:02.883078 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:02.883088 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:02.883092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:02.883263 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:02.883652 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:02.883657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:02.883661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:02.884093 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:02.884403 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:02.884714 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:52:02.884930 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:52:02.886010 7649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:03.001842 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 7518
I20260927 22:52:03.021256 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 7667
I20260927 22:52:03.033814 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:03.022105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.0.97.3:35043)
0927 22:52:03.022265 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:03.022269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:03.022285 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:52:03.022363 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:03.022367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:03.022422 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:03.022512 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:03.023777 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:03.024502 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:03.025839 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:03.026242 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:03.026345 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:03.028713 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:03.028725 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:03.028741 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:03.028768 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:03.031478 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:03.032050 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:03.032056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:03.032058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:03.032164 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:03.032546 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:03.032549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:03.032551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:03.032864 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:03.033071 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:03.033299 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 22:52:03.033451 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:52:03.034431 7790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:03.132203 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:03.131458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:36000)
0927 22:52:03.131946 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:03.132040 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":302,"threads_started":1}
I20260927 22:52:03.133445 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:03.133106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35383 (local address 127.0.0.1:36010)
0927 22:52:03.133256 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:03.133337 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:52:03.150067 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 7667
I20260927 22:52:03.171763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 7002
I20260927 22:52:03.172853 7168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:03.285398 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 7002
W20260927 22:52:03.312548 7312 connection.cc:535] client connection to 127.0.97.61:38543 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:52:03.312804 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 7101
I20260927 22:52:03.313822 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:03.313321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38543 (local address 127.0.0.1:40630)
0927 22:52:03.313515 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:03.313647 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 22:52:03.313941 7357 generic_service.cc:196] Checking for leaks (request via RPC)
W20260927 22:52:03.430790 7312 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2a563337cda04b02bde7e5909c712343 -> Peer 2a73ae77b7f54c929293c60d37f40189 (127.0.97.61:38543): Couldn't send request to peer 2a73ae77b7f54c929293c60d37f40189. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260927 22:52:03.430989 7312 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.97.61:38543, user_credentials={real_user=slave}} blocked reactor thread for 116944us
I20260927 22:52:03.431272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 7101
2026-09-27T22:52:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10945 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:03 dist-test-slave-df53 krb5kdc[7935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:03 dist-test-slave-df53 krb5kdc[7935](info): set up 2 sockets
Sep 27 22:52:03 dist-test-slave-df53 krb5kdc[7935](info): commencing operation
krb5kdc: starting...
W20260927 22:52:04.526605 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:04 dist-test-slave-df53 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 22:52:04.733902 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40851
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:05.039848 7948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:05.040182 7948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:05.040333 7948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:05.047426 7948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:05.047561 7948 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:52:05.047652 7948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:05.047775 7948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:05.047858 7948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:05.047945 7948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:05.057365 7948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40851
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:05.058605 7948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:05.059850 7948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:05.071173 7956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:05.071193 7954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:05.071593 7953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:05.071867 7948 server_base.cc:1034] running on GCE node
I20260927 22:52:05.072782 7948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:05.074327 7948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:05.075603 7948 hybrid_clock.cc:648] HybridClock initialized: now 1790549525075521 us; error 51 us; skew 500 ppm
I20260927 22:52:05.076048 7948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:05 dist-test-slave-df53 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:05.079632 7948 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 22:52:05.081004 7948 webserver.cc:460] Webserver started at http://127.0.97.62:39891/ using document root <none> and password file <none>
I20260927 22:52:05.081736 7948 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:05.081856 7948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:05.082185 7948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:05.084940 7948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "b165fd345adb45f58a4879382716d1c9"
format_stamp: "Formatted at 2026-09-27 22:52:05 on dist-test-slave-df53"
server_key: "13d7ba3d9edf6ac08ce67f645eaed582"
server_key_iv: "2989c1cbfdff65963a49d45c4fd32ef3"
server_key_version: "encryptionkey@0"
I20260927 22:52:05.085774 7948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "b165fd345adb45f58a4879382716d1c9"
format_stamp: "Formatted at 2026-09-27 22:52:05 on dist-test-slave-df53"
server_key: "13d7ba3d9edf6ac08ce67f645eaed582"
server_key_iv: "2989c1cbfdff65963a49d45c4fd32ef3"
server_key_version: "encryptionkey@0"
I20260927 22:52:05.090512 7948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:52:05.093740 7963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:05.094913 7948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:52:05.095131 7948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "b165fd345adb45f58a4879382716d1c9"
format_stamp: "Formatted at 2026-09-27 22:52:05 on dist-test-slave-df53"
server_key: "13d7ba3d9edf6ac08ce67f645eaed582"
server_key_iv: "2989c1cbfdff65963a49d45c4fd32ef3"
server_key_version: "encryptionkey@0"
I20260927 22:52:05.095357 7948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:05.110108 7948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:05.113829 7948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:05.114082 7948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:05.134775 7948 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40851
I20260927 22:52:05.134795 8024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40851 every 8 connection(s)
I20260927 22:52:05.136054 7948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:05.139635 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:05.140924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 7948
I20260927 22:52:05.141131 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:05.141970 388 external_mini_cluster.cc:1428] Setting key 39fd9017b4f540eaa6cc554e7484ffa8
I20260927 22:52:05.152079 8025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9: Bootstrap starting.
Sep 27 22:52:05 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:52:05.155213 8025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:05.156195 8025 log.cc:824] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:05.158947 8025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9: No bootstrap required, opened a new log
I20260927 22:52:05.162406 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:05.143943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.0.97.62:40851)
0927 22:52:05.144677 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:05.144688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:05.144740 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:52:05.146339 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:05.146359 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:05.146722 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:05.147084 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:05.148409 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:05.149715 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:05.150928 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:05.151564 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:05.151748 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:05.155720 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:05.155727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:05.155739 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:05.155769 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:05.158714 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:05.159394 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:05.159401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:05.159405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:05.159556 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:05.159913 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:05.159919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:05.159922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:05.160398 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:05.160691 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:05.161095 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 22:52:05.161471 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":328,"threads_started":1}
I20260927 22:52:05.166898 8025 raft_consensus.cc:348] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165fd345adb45f58a4879382716d1c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40851 } }
I20260927 22:52:05.167222 8025 raft_consensus.cc:374] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:05.167321 8025 raft_consensus.cc:729] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b165fd345adb45f58a4879382716d1c9, State: Initialized, Role: FOLLOWER
I20260927 22:52:05.167825 8025 consensus_queue.cc:260] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165fd345adb45f58a4879382716d1c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40851 } }
I20260927 22:52:05.168051 8025 raft_consensus.cc:388] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:05.168190 8025 raft_consensus.cc:482] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:05.168349 8025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:05.170627 8025 raft_consensus.cc:504] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165fd345adb45f58a4879382716d1c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40851 } }
I20260927 22:52:05.171068 8025 leader_election.cc:302] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b165fd345adb45f58a4879382716d1c9; no voters:
I20260927 22:52:05.171577 8025 leader_election.cc:288] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:05.171690 8030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:05.172432 8030 raft_consensus.cc:686] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [term 1 LEADER]: Becoming Leader. State: Replica: b165fd345adb45f58a4879382716d1c9, State: Running, Role: LEADER
I20260927 22:52:05.172899 8030 consensus_queue.cc:237] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165fd345adb45f58a4879382716d1c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40851 } }
I20260927 22:52:05.173434 8025 sys_catalog.cc:564] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:05.175920 8031 sys_catalog.cc:455] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b165fd345adb45f58a4879382716d1c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165fd345adb45f58a4879382716d1c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40851 } } }
I20260927 22:52:05.176170 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b165fd345adb45f58a4879382716d1c9. Latest consensus state: current_term: 1 leader_uuid: "b165fd345adb45f58a4879382716d1c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b165fd345adb45f58a4879382716d1c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40851 } } }
I20260927 22:52:05.176345 8031 sys_catalog.cc:458] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:05.176404 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:05.177290 8037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:05.181643 8037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:05.189852 8037 catalog_manager.cc:1347] Generated new cluster ID: 8ebbf7b3dc0a4f73a4ccd21a9b0efd45
I20260927 22:52:05.189954 8037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:05.199159 8037 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:05.200822 8037 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:05.210861 8037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9: Generated new TSK 0
I20260927 22:52:05.211795 8037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 22:52:05.265918 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40851
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:05.587841 8051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:05.588130 8051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:05.588356 8051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:05.595942 8051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:05.596029 8051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:05.596123 8051 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:52:05.596199 8051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:52:05.606231 8051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:05.607271 8051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:05.608455 8051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:05.620819 8057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:05.620911 8059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:05.620879 8056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:05.621507 8051 server_base.cc:1034] running on GCE node
I20260927 22:52:05.622071 8051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:05.622890 8051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:05.624128 8051 hybrid_clock.cc:648] HybridClock initialized: now 1790549525624055 us; error 51 us; skew 500 ppm
I20260927 22:52:05.624439 8051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:05 dist-test-slave-df53 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:05.627501 8051 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:52:05.628829 8051 webserver.cc:460] Webserver started at http://127.0.97.1:38329/ using document root <none> and password file <none>
I20260927 22:52:05.629447 8051 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:05.629552 8051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:05.629897 8051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:05.632486 8051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "b514f9b69fce46fe956bf2be257d729f"
format_stamp: "Formatted at 2026-09-27 22:52:05 on dist-test-slave-df53"
server_key: "326ff7cb84831338cab2ad32c4667e22"
server_key_iv: "f8f57480e3a10179f31057b6a5e1252f"
server_key_version: "encryptionkey@0"
I20260927 22:52:05.633239 8051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b514f9b69fce46fe956bf2be257d729f"
format_stamp: "Formatted at 2026-09-27 22:52:05 on dist-test-slave-df53"
server_key: "326ff7cb84831338cab2ad32c4667e22"
server_key_iv: "f8f57480e3a10179f31057b6a5e1252f"
server_key_version: "encryptionkey@0"
I20260927 22:52:05.637933 8051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260927 22:52:05.640916 8066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:05.642169 8051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:52:05.642326 8051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "b514f9b69fce46fe956bf2be257d729f"
format_stamp: "Formatted at 2026-09-27 22:52:05 on dist-test-slave-df53"
server_key: "326ff7cb84831338cab2ad32c4667e22"
server_key_iv: "f8f57480e3a10179f31057b6a5e1252f"
server_key_version: "encryptionkey@0"
I20260927 22:52:05.642537 8051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:05.665359 8051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:05.669036 8051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:05.669308 8051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:05.671880 8051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:05.672045 8051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 22:52:05.672217 8051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:05.672327 8051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:52:05 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:52:05.698354 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:05.672164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41755 (local address 127.0.97.62:40851)
0927 22:52:05.672622 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:05.672628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:05.673751 (+ 1123us) server_negotiation.cc:407] Connection header received
0927 22:52:05.675893 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:05.675899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:05.675988 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:05.676105 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:05.678405 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:05.679677 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:05.682958 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:05.683688 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:05.683836 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:05.689361 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:05.689371 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:05.689379 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:05.689428 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:05.693139 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:05.694331 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:05.694336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:05.694339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:05.694485 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:05.695095 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:05.695103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:05.695108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:05.695488 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:05.697328 (+ 1840us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:05.697509 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 22:52:05.697828 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":239,"threads_started":1}
I20260927 22:52:05.698459 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:05.672772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40851 (local address 127.0.97.1:41755)
0927 22:52:05.673578 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:05.673644 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:05.675624 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:05.676322 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:05.676329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:05.677040 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:05.678127 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:05.678165 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:05.679908 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:05.679916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:05.682757 (+ 2841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:05.682777 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:05.683982 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:05.683992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:05.684428 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:05.685232 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:05.685276 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:05.689068 (+ 3792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:05.693310 (+ 4242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:05.693321 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:05.693346 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:05.694113 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:05.694628 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:05.694634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:05.694643 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:05.694942 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:05.695660 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:05.695668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:05.696203 (+ 535us) client_negotiation.cc:769] Sending connection context
0927 22:52:05.696568 (+ 365us) client_negotiation.cc:240] Negotiation successful
0927 22:52:05.697289 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":342,"threads_started":1}
I20260927 22:52:05.711455 8051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38097
I20260927 22:52:05.711462 8186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38097 every 8 connection(s)
I20260927 22:52:05.712842 8051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:52:05.714911 8187 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40851
I20260927 22:52:05.715124 8187 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:05.715853 8187 heartbeater.cc:511] Master 127.0.97.62:40851 requested a full tablet report, sending...
I20260927 22:52:05.715889 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 8051
I20260927 22:52:05.716078 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:52:05.717156 388 external_mini_cluster.cc:1428] Setting key 1845dde1aea93912e0988718ee4c5408
I20260927 22:52:05.718098 7980 ts_manager.cc:194] Registered new tserver with Master: b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097)
I20260927 22:52:05.719514 7980 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41755
I20260927 22:52:05.720081 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:05.705663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44649 (local address 127.0.97.62:40851)
0927 22:52:05.705864 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:05.705872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:05.706718 (+ 846us) server_negotiation.cc:407] Connection header received
0927 22:52:05.706917 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:05.706925 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:05.707009 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:05.707111 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:05.708740 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:05.709660 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:05.711921 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:05.712321 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:05.712412 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:05.714638 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:05.714644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:05.714650 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:05.714683 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:05.717513 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:05.718212 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:05.718218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:05.718222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:05.718356 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:05.718705 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:05.718711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:05.718715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:05.718985 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:05.719140 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:05.719534 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 22:52:05.719733 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:52:05.720122 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:05.705748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40851 (local address 127.0.97.1:44649)
0927 22:52:05.706617 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:05.706631 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:05.706791 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:05.707185 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:05.707193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:05.708003 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:05.708558 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:05.708580 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:05.709787 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:05.709792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:05.711796 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:05.711808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:05.712500 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:05.712506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:05.712663 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:05.713328 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:05.713345 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:05.714465 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:05.717658 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:05.717665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:05.717671 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:05.718092 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:05.718469 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:05.718474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:05.718477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:05.718601 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:05.719105 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:05.719109 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:05.719279 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 22:52:05.719426 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 22:52:05.719699 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":519,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 22:52:05.765074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40851
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:06.071420 8191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:06.071681 8191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:06.071828 8191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:06.078971 8191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:06.079068 8191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:06.079236 8191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:52:06.079346 8191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:52:06.088732 8191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:06.089926 8191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:06.091176 8191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:06.103470 8197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:06.103543 8196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:06.103570 8199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:06.104776 8191 server_base.cc:1034] running on GCE node
I20260927 22:52:06.105300 8191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:06.106172 8191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:06.107378 8191 hybrid_clock.cc:648] HybridClock initialized: now 1790549526107328 us; error 36 us; skew 500 ppm
I20260927 22:52:06.107607 8191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:06 dist-test-slave-df53 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:06.110698 8191 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:52:06.112082 8191 webserver.cc:460] Webserver started at http://127.0.97.2:42019/ using document root <none> and password file <none>
I20260927 22:52:06.112744 8191 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:06.112856 8191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:06.113200 8191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:06.116003 8191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "7e202f17d16e44d1bf4fff4c32a23769"
format_stamp: "Formatted at 2026-09-27 22:52:06 on dist-test-slave-df53"
server_key: "98bfdd7fa5c3d81c89df28cb77705849"
server_key_iv: "5a33d3de0355b81175ba1e0117558e03"
server_key_version: "encryptionkey@0"
I20260927 22:52:06.116744 8191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7e202f17d16e44d1bf4fff4c32a23769"
format_stamp: "Formatted at 2026-09-27 22:52:06 on dist-test-slave-df53"
server_key: "98bfdd7fa5c3d81c89df28cb77705849"
server_key_iv: "5a33d3de0355b81175ba1e0117558e03"
server_key_version: "encryptionkey@0"
I20260927 22:52:06.121444 8191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 22:52:06.124801 8206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:06.126122 8191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:06.126278 8191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "7e202f17d16e44d1bf4fff4c32a23769"
format_stamp: "Formatted at 2026-09-27 22:52:06 on dist-test-slave-df53"
server_key: "98bfdd7fa5c3d81c89df28cb77705849"
server_key_iv: "5a33d3de0355b81175ba1e0117558e03"
server_key_version: "encryptionkey@0"
I20260927 22:52:06.126489 8191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:06.157142 8191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:06.160769 8191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:06.161012 8191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:06.163564 8191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:06.163669 8191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:06.163798 8191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:06.163877 8191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:52:06 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:52:06.184994 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.164177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45387 (local address 127.0.97.62:40851)
0927 22:52:06.164373 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.164381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.165795 (+ 1414us) server_negotiation.cc:407] Connection header received
0927 22:52:06.167724 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.167730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.167823 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.167925 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:06.169918 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.170893 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.173403 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.173923 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.174076 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.178856 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:06.178862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:06.178865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:06.178901 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:06.181665 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:06.182710 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:06.182716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:06.182722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:06.182876 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:06.183315 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:06.183324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:06.183326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:06.183589 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:06.183769 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.184435 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.184647 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:52:06.186254 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.164837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40851 (local address 127.0.97.2:45387)
0927 22:52:06.165580 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:06.165685 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:06.167452 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:06.168140 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:06.168147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:06.168848 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:06.169705 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.169736 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.171087 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.171093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.173216 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.173239 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.174216 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.174225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.174416 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.175113 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:06.175153 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:06.178601 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:06.181807 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:06.181816 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:06.181831 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:06.182562 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:06.182986 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:06.182992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:06.182996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:06.183192 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:06.183743 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:06.183750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:06.184087 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 22:52:06.184339 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:52:06.184821 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":333,"threads_started":1}
I20260927 22:52:06.209988 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.196192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56215 (local address 127.0.97.62:40851)
0927 22:52:06.196380 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.196386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.197116 (+ 730us) server_negotiation.cc:407] Connection header received
0927 22:52:06.197307 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.197311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.197370 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.197447 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:06.199293 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.200350 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.202411 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.202801 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.202921 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.204712 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:06.204719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:06.204722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:06.204751 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:06.207222 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:06.208080 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:06.208087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:06.208092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:06.208229 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:06.208620 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:06.208627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:06.208630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:06.208925 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:06.209088 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.209434 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.209671 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:52:06.210101 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.196262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40851 (local address 127.0.97.2:56215)
0927 22:52:06.196990 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:06.197015 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:06.197175 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:06.197508 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:06.197516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:06.198374 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:06.199110 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.199131 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.200482 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.200487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.202274 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.202285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.202977 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.202984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.203161 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.203843 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:06.203858 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:06.204508 (+ 650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:06.207365 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:06.207371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:06.207376 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:06.207897 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:06.208361 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:06.208367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:06.208376 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:06.208509 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:06.209048 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:06.209053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:06.209212 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 22:52:06.209357 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 22:52:06.209671 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":367,"threads_started":1}
I20260927 22:52:06.223227 8191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39881
I20260927 22:52:06.223268 8325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39881 every 8 connection(s)
I20260927 22:52:06.224517 8191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:52:06.226449 8326 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40851
I20260927 22:52:06.226663 8326 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:06.227268 8326 heartbeater.cc:511] Master 127.0.97.62:40851 requested a full tablet report, sending...
I20260927 22:52:06.228506 7979 ts_manager.cc:194] Registered new tserver with Master: 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881)
I20260927 22:52:06.229355 7979 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45387
I20260927 22:52:06.233505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 8191
I20260927 22:52:06.233686 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:52:06.234144 388 external_mini_cluster.cc:1428] Setting key b295f7558fe9f236a3f502e15d5a7263
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 22:52:06.276094 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40851
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:06.586361 8330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:06.586621 8330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:06.586804 8330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:06.594017 8330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:06.594130 8330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:06.594281 8330 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:52:06.594398 8330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:52:06.604029 8330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:06.605190 8330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:06.606539 8330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:06.618810 8335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:06.618845 8336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:06.619074 8338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:06.619620 8330 server_base.cc:1034] running on GCE node
I20260927 22:52:06.620220 8330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:06.621095 8330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:06.622334 8330 hybrid_clock.cc:648] HybridClock initialized: now 1790549526622278 us; error 52 us; skew 500 ppm
I20260927 22:52:06.622633 8330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:06 dist-test-slave-df53 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:06.625521 8330 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:52:06.626987 8330 webserver.cc:460] Webserver started at http://127.0.97.3:33595/ using document root <none> and password file <none>
I20260927 22:52:06.627660 8330 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:06.627789 8330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:06.628100 8330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:06.631217 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "3e52b061730e43ef9b024497962ea92b"
format_stamp: "Formatted at 2026-09-27 22:52:06 on dist-test-slave-df53"
server_key: "b7cfdac298560aa2369cf80ef403da6d"
server_key_iv: "b87e34faf67848134188a721907df0b9"
server_key_version: "encryptionkey@0"
I20260927 22:52:06.632124 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3e52b061730e43ef9b024497962ea92b"
format_stamp: "Formatted at 2026-09-27 22:52:06 on dist-test-slave-df53"
server_key: "b7cfdac298560aa2369cf80ef403da6d"
server_key_iv: "b87e34faf67848134188a721907df0b9"
server_key_version: "encryptionkey@0"
I20260927 22:52:06.637101 8330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:52:06.640385 8345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:06.641561 8330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:52:06.641774 8330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "3e52b061730e43ef9b024497962ea92b"
format_stamp: "Formatted at 2026-09-27 22:52:06 on dist-test-slave-df53"
server_key: "b7cfdac298560aa2369cf80ef403da6d"
server_key_iv: "b87e34faf67848134188a721907df0b9"
server_key_version: "encryptionkey@0"
I20260927 22:52:06.641983 8330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:06.660444 8330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:06.664496 8330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:06.664760 8330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:06.667273 8330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:06.667395 8330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:06.667526 8330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:06.667624 8330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:52:06 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:52:06.691314 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.667631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45857 (local address 127.0.97.62:40851)
0927 22:52:06.667832 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.667841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.669122 (+ 1281us) server_negotiation.cc:407] Connection header received
0927 22:52:06.671091 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.671097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.671182 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.671293 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:06.673352 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.674468 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.676938 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.677538 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.677711 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.682102 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:06.682112 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:06.682119 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:06.682157 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:06.686333 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:06.687275 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:06.687285 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:06.687291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:06.687460 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:06.688003 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:06.688011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:06.688017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:06.688410 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:06.688556 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.690849 (+ 2293us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.691076 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:52:06.691975 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.668132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40851 (local address 127.0.97.3:45857)
0927 22:52:06.668955 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:06.669001 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:06.670828 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:06.671496 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:06.671504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:06.672193 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:06.673107 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.673137 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.674617 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.674623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.676772 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.676787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.677838 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.677846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.678196 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.678917 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:06.678955 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:06.681870 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:06.686483 (+ 4613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:06.686489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:06.686500 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:06.687117 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:06.687611 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:06.687618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:06.687624 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:06.687886 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:06.688550 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:06.688558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:06.689018 (+ 460us) client_negotiation.cc:769] Sending connection context
0927 22:52:06.689408 (+ 390us) client_negotiation.cc:240] Negotiation successful
0927 22:52:06.689973 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":321,"threads_started":1}
I20260927 22:52:06.713267 8330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40797
I20260927 22:52:06.713284 8464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40797 every 8 connection(s)
I20260927 22:52:06.714854 8330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:52:06.718878 8465 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40851
I20260927 22:52:06.719106 8465 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:06.719882 8465 heartbeater.cc:511] Master 127.0.97.62:40851 requested a full tablet report, sending...
I20260927 22:52:06.720005 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.699613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42637 (local address 127.0.97.62:40851)
0927 22:52:06.699805 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.699813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.703314 (+ 3501us) server_negotiation.cc:407] Connection header received
0927 22:52:06.703503 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.703511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.703626 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.703732 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:06.705091 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.705997 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.708591 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.709355 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.709502 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.714663 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:06.714670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:06.714674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:06.714709 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:06.717410 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:06.718142 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:06.718151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:06.718154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:06.718278 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:06.718707 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:06.718715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:06.718721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:06.719050 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:06.719178 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.719617 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.719770 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:52:06.720311 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.702247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40851 (local address 127.0.97.3:42637)
0927 22:52:06.703226 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:06.703246 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:06.703380 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:06.703791 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:06.703799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:06.704525 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:06.704938 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.704952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.706143 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.706152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.708431 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.708445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.709586 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.709624 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.709863 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.713760 (+ 3897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:06.713782 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:06.714481 (+ 699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:06.717527 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:06.717534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:06.717537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:06.718028 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:06.718390 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:06.718395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:06.718398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:06.718567 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:06.719157 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:06.719162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:06.719352 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 22:52:06.719516 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 22:52:06.719823 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":582,"threads_started":1}
I20260927 22:52:06.721426 7980 ts_manager.cc:194] Registered new tserver with Master: 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797)
I20260927 22:52:06.722244 7980 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45857
I20260927 22:52:06.722867 8187 heartbeater.cc:503] Master 127.0.97.62:40851 was elected leader, sending a full tablet report...
I20260927 22:52:06.723842 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 8330
I20260927 22:52:06.724017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:52:06.724409 388 external_mini_cluster.cc:1428] Setting key 9de5f0e8b27c20881cb6d224de29f047
I20260927 22:52:06.727074 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:52:06 dist-test-slave-df53 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:52:06 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:06 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:52:06.772240 8074 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:06.762540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.0.97.62:40851)
0927 22:52:06.762704 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.762708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.763056 (+ 348us) server_negotiation.cc:407] Connection header received
0927 22:52:06.763202 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.763206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.763268 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.763354 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:06.764582 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.765311 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.766579 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.766967 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.767068 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.771916 (+ 4848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39136: BlockingRecv error: recv got EOF from 127.0.0.1:39136 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 22:52:06 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:52:06.787287 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.774804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.0.97.62:40851)
0927 22:52:06.775025 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.775030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.775198 (+ 168us) server_negotiation.cc:407] Connection header received
0927 22:52:06.775312 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.775315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.775380 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.775455 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:06.776462 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.777184 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.778359 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.778766 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.778859 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.782377 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:06.782382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:06.782385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:06.782414 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:06.784802 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:06.785427 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:06.785434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:06.785437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:06.785556 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:06.785983 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:06.785989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:06.785992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:06.786247 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:06.786532 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.786777 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.786991 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:52:06.791146 7979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:52:06.798003 7979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:52:06.832962 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.810198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.97.3:40797)
0927 22:52:06.823122 (+ 12924us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.823134 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.823176 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:52:06.823287 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.823298 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.823618 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.823845 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:06.825719 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.827415 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.830785 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.831853 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.832003 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.832250 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.832470 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.832629 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12713,"thread_start_us":12505,"threads_started":1}
I20260927 22:52:06.833405 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.810163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39881 (local address 127.0.0.1:49084)
0927 22:52:06.824585 (+ 14422us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:06.824607 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:06.824793 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:06.825960 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:06.825966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:06.825993 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:06.826374 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.826387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.828328 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.828333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.830503 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.830514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.832285 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.832292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.832458 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.832492 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:52:06.832552 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 22:52:06.832670 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14273,"mutex_wait_us":144,"thread_start_us":12344,"threads_started":1}
I20260927 22:52:06.833891 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.810828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.0.97.2:39881)
0927 22:52:06.825291 (+ 14463us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.825301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.825331 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:52:06.825454 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.825464 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.825715 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.826563 (+ 848us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:06.826782 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.828186 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.830702 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.832056 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.833165 (+ 1109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.833346 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.833416 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.833571 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14327,"thread_start_us":14205,"threads_started":1}
I20260927 22:52:06.833917 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.810757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.0.97.1:38097)
0927 22:52:06.824515 (+ 13758us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.824527 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.825815 (+ 1288us) server_negotiation.cc:407] Connection header received
0927 22:52:06.825969 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.825975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.826170 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.826353 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:06.827155 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.828211 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.830983 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.832338 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.832451 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.832651 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.833447 (+ 796us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.833639 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13578,"thread_start_us":13751,"threads_started":1}
I20260927 22:52:06.834187 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.809820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40797 (local address 127.0.0.1:60392)
0927 22:52:06.822037 (+ 12217us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:06.822083 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:06.822283 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:06.824845 (+ 2562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:06.824857 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:06.824901 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:06.825406 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.825420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.827555 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.827571 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.830608 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.830619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.832041 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.832047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.832226 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.832282 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:52:06.832431 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:52:06.832612 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11887,"mutex_wait_us":103,"thread_start_us":14455,"threads_started":1}
I20260927 22:52:06.834450 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.810544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38097 (local address 127.0.0.1:51582)
0927 22:52:06.825683 (+ 15139us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:06.825699 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:06.825820 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:06.826543 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:06.826548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:06.826570 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:06.826962 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.826973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.828394 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.828401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.830862 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.830875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.832820 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.832826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.832997 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.833033 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:52:06.833125 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:52:06.833229 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15011,"mutex_wait_us":52,"thread_start_us":14865,"threads_started":1}
I20260927 22:52:06.837510 8395 tablet_service.cc:1467] Processing CreateTablet for tablet ca00f74dbf4d48b69e70cd1176271880 (DEFAULT_TABLE table=test-table [id=bd4dd53e699d4ab9be118f7b922e7531]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:06.837673 8117 tablet_service.cc:1467] Processing CreateTablet for tablet ca00f74dbf4d48b69e70cd1176271880 (DEFAULT_TABLE table=test-table [id=bd4dd53e699d4ab9be118f7b922e7531]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:06.837523 8260 tablet_service.cc:1467] Processing CreateTablet for tablet ca00f74dbf4d48b69e70cd1176271880 (DEFAULT_TABLE table=test-table [id=bd4dd53e699d4ab9be118f7b922e7531]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:06.838754 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca00f74dbf4d48b69e70cd1176271880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:06.838790 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca00f74dbf4d48b69e70cd1176271880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:06.838800 8260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca00f74dbf4d48b69e70cd1176271880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:06.849639 8489 tablet_bootstrap.cc:492] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: Bootstrap starting.
I20260927 22:52:06.852787 8489 tablet_bootstrap.cc:654] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:06.853291 8490 tablet_bootstrap.cc:492] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: Bootstrap starting.
I20260927 22:52:06.854144 8489 log.cc:824] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:06.855513 8491 tablet_bootstrap.cc:492] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: Bootstrap starting.
I20260927 22:52:06.856660 8490 tablet_bootstrap.cc:654] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:06.857103 8489 tablet_bootstrap.cc:492] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: No bootstrap required, opened a new log
I20260927 22:52:06.857355 8489 ts_tablet_manager.cc:1397] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 22:52:06.857872 8490 log.cc:824] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:06.858808 8491 tablet_bootstrap.cc:654] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:06.859965 8491 log.cc:824] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:06.860838 8490 tablet_bootstrap.cc:492] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: No bootstrap required, opened a new log
I20260927 22:52:06.861186 8490 ts_tablet_manager.cc:1397] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:52:06.864939 8489 raft_consensus.cc:348] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } }
I20260927 22:52:06.865293 8489 raft_consensus.cc:374] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:06.865442 8489 raft_consensus.cc:729] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e52b061730e43ef9b024497962ea92b, State: Initialized, Role: FOLLOWER
I20260927 22:52:06.866139 8489 consensus_queue.cc:260] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } }
I20260927 22:52:06.867204 8465 heartbeater.cc:503] Master 127.0.97.62:40851 was elected leader, sending a full tablet report...
I20260927 22:52:06.867694 8489 ts_tablet_manager.cc:1428] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 22:52:06.867772 8491 tablet_bootstrap.cc:492] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: No bootstrap required, opened a new log
I20260927 22:52:06.868065 8491 ts_tablet_manager.cc:1397] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260927 22:52:06.872988 8490 raft_consensus.cc:348] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } }
I20260927 22:52:06.873442 8490 raft_consensus.cc:374] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:06.873555 8490 raft_consensus.cc:729] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e202f17d16e44d1bf4fff4c32a23769, State: Initialized, Role: FOLLOWER
I20260927 22:52:06.874090 8490 consensus_queue.cc:260] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } }
I20260927 22:52:06.875083 8326 heartbeater.cc:503] Master 127.0.97.62:40851 was elected leader, sending a full tablet report...
I20260927 22:52:06.875623 8490 ts_tablet_manager.cc:1428] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 22:52:06.876647 8491 raft_consensus.cc:348] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } }
I20260927 22:52:06.876974 8491 raft_consensus.cc:374] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:06.877076 8491 raft_consensus.cc:729] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b514f9b69fce46fe956bf2be257d729f, State: Initialized, Role: FOLLOWER
I20260927 22:52:06.877641 8491 consensus_queue.cc:260] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } }
I20260927 22:52:06.879074 8491 ts_tablet_manager.cc:1428] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 22:52:06.937446 8495 raft_consensus.cc:482] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:52:06.937815 8495 raft_consensus.cc:504] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } }
I20260927 22:52:06.939558 8495 leader_election.cc:288] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881), b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097)
I20260927 22:52:06.946206 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.939934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44969 (local address 127.0.97.2:39881)
0927 22:52:06.940106 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.940112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.940133 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:52:06.940204 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.940209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.940316 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.940412 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:06.941043 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.942046 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.944254 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.945257 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.945384 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.945490 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.945807 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.945931 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 22:52:06.946259 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.939740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39881 (local address 127.0.97.3:44969)
0927 22:52:06.939948 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:06.939970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:06.940092 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:06.940595 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:06.940600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:06.940634 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:06.940884 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.940897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.942176 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.942180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.944115 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.944129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.945448 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.945457 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.945632 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.945671 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:52:06.945755 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:52:06.945927 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":32}
I20260927 22:52:06.946954 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.940440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51351 (local address 127.0.97.1:38097)
0927 22:52:06.940639 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.940643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.940828 (+ 185us) server_negotiation.cc:407] Connection header received
0927 22:52:06.941029 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.941034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.941119 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.941223 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:06.941768 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.942581 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.944703 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.945978 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.946155 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.946254 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.946626 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.946768 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:52:06.947213 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.939955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38097 (local address 127.0.97.3:51351)
0927 22:52:06.940731 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:06.940746 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:06.940863 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:06.941240 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:06.941243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:06.941262 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:06.941560 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.941568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.942713 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.942720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.944563 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:06.944573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.946226 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.946232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:06.946438 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.946479 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:52:06.946605 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 22:52:06.946760 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":5,"thread_start_us":430,"threads_started":1}
I20260927 22:52:06.947505 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca00f74dbf4d48b69e70cd1176271880" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e202f17d16e44d1bf4fff4c32a23769" is_pre_election: true
I20260927 22:52:06.947963 8281 raft_consensus.cc:2454] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 0.
I20260927 22:52:06.948594 8347 leader_election.cc:302] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e52b061730e43ef9b024497962ea92b, 7e202f17d16e44d1bf4fff4c32a23769; no voters:
I20260927 22:52:06.949036 8495 raft_consensus.cc:2781] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:52:06.949186 8495 raft_consensus.cc:482] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:52:06.949290 8495 raft_consensus.cc:3037] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:06.949110 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca00f74dbf4d48b69e70cd1176271880" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b514f9b69fce46fe956bf2be257d729f" is_pre_election: true
I20260927 22:52:06.949647 8137 raft_consensus.cc:2454] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 0.
I20260927 22:52:06.951647 8495 raft_consensus.cc:504] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } }
I20260927 22:52:06.952214 8495 leader_election.cc:288] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 election: Requested vote from peers 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881), b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097)
I20260927 22:52:06.952618 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca00f74dbf4d48b69e70cd1176271880" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e202f17d16e44d1bf4fff4c32a23769"
I20260927 22:52:06.952662 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca00f74dbf4d48b69e70cd1176271880" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b514f9b69fce46fe956bf2be257d729f"
I20260927 22:52:06.952827 8281 raft_consensus.cc:3037] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:06.952832 8137 raft_consensus.cc:3037] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:06.954936 8137 raft_consensus.cc:2454] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 1.
I20260927 22:52:06.955081 8281 raft_consensus.cc:2454] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 1.
I20260927 22:52:06.955353 8349 leader_election.cc:302] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e52b061730e43ef9b024497962ea92b, b514f9b69fce46fe956bf2be257d729f; no voters:
I20260927 22:52:06.955667 8495 raft_consensus.cc:2781] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:06.956030 8495 raft_consensus.cc:686] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 1 LEADER]: Becoming Leader. State: Replica: 3e52b061730e43ef9b024497962ea92b, State: Running, Role: LEADER
I20260927 22:52:06.956558 8495 consensus_queue.cc:237] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } }
I20260927 22:52:06.960731 7979 catalog_manager.cc:5495] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e52b061730e43ef9b024497962ea92b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3e52b061730e43ef9b024497962ea92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } health_report { overall_health: UNKNOWN } } }
W20260927 22:52:06.965224 8188 tablet.cc:2369] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:52:06.966984 8466 tablet.cc:2369] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:52:06.976382 8327 tablet.cc:2369] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:52:06.984500 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:06.979634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.0.97.3:40797)
0927 22:52:06.979810 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:06.979815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:06.979831 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:52:06.980012 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:06.980015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:06.980083 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:06.980206 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:06.980482 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.981164 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.982721 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:06.983137 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:06.983230 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:06.983586 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:06.984060 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:06.984169 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:06.984246 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 22:52:06.984301 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:52:06.992115 8281 raft_consensus.cc:1261] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Refusing update from remote peer 3e52b061730e43ef9b024497962ea92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:06.992151 8137 raft_consensus.cc:1261] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Refusing update from remote peer 3e52b061730e43ef9b024497962ea92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:06.993009 8499 consensus_queue.cc:1035] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:06.993366 8495 consensus_queue.cc:1035] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:07.003677 8503 mvcc.cc:205] Tried to move back new op lower bound from 7334090862550921216 to 7334090862417547264. Current Snapshot: MvccSnapshot[applied={T|T < 7334090862550921216}]
I20260927 22:52:07.006000 8504 mvcc.cc:205] Tried to move back new op lower bound from 7334090862550921216 to 7334090862417547264. Current Snapshot: MvccSnapshot[applied={T|T < 7334090862550921216}]
I20260927 22:52:07.007542 8502 mvcc.cc:205] Tried to move back new op lower bound from 7334090862550921216 to 7334090862417547264. Current Snapshot: MvccSnapshot[applied={T|T < 7334090862550921216 or (T in {7334090862550921216})}]
I20260927 22:52:07.015951 7979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:07.016175 7979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:07.019327 7979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9: Sending DeleteTablet for 3 replicas of tablet ca00f74dbf4d48b69e70cd1176271880
I20260927 22:52:07.020442 8260 tablet_service.cc:1514] Processing DeleteTablet for tablet ca00f74dbf4d48b69e70cd1176271880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49084
I20260927 22:52:07.020416 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet ca00f74dbf4d48b69e70cd1176271880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60392
I20260927 22:52:07.021389 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet ca00f74dbf4d48b69e70cd1176271880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51582
I20260927 22:52:07.021423 7979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:52:07.022076 7979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:52:07.024627 8513 tablet_replica.cc:330] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: stopping tablet replica
I20260927 22:52:07.025058 8513 raft_consensus.cc:2229] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:07.025501 8513 raft_consensus.cc:2258] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:07.026268 8511 tablet_replica.cc:330] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: stopping tablet replica
I20260927 22:52:07.026674 8511 raft_consensus.cc:2229] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:07.027141 8511 raft_consensus.cc:2258] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:07.027875 8512 tablet_replica.cc:330] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: stopping tablet replica
I20260927 22:52:07.028270 8512 raft_consensus.cc:2229] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:52:07.029119 8511 ts_tablet_manager.cc:1905] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:07.029733 8513 ts_tablet_manager.cc:1905] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:07.030350 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 8230e09cc08b479d9369002bb38b4bd8 (DEFAULT_TABLE table=test-table [id=ef26e2cf68d24008bd3de4acc27e6a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:07.030754 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8230e09cc08b479d9369002bb38b4bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:07.034768 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 8230e09cc08b479d9369002bb38b4bd8 (DEFAULT_TABLE table=test-table [id=ef26e2cf68d24008bd3de4acc27e6a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:07.035249 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8230e09cc08b479d9369002bb38b4bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:07.036062 8511 ts_tablet_manager.cc:1918] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:07.036206 8511 log.cc:1196] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/ca00f74dbf4d48b69e70cd1176271880
I20260927 22:52:07.036501 8513 ts_tablet_manager.cc:1918] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:07.036633 8513 log.cc:1196] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/ca00f74dbf4d48b69e70cd1176271880
I20260927 22:52:07.036700 8511 ts_tablet_manager.cc:1939] T ca00f74dbf4d48b69e70cd1176271880 P 7e202f17d16e44d1bf4fff4c32a23769: Deleting consensus metadata
I20260927 22:52:07.037129 8513 ts_tablet_manager.cc:1939] T ca00f74dbf4d48b69e70cd1176271880 P b514f9b69fce46fe956bf2be257d729f: Deleting consensus metadata
I20260927 22:52:07.028896 8512 raft_consensus.cc:2258] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:07.038062 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 8230e09cc08b479d9369002bb38b4bd8 (DEFAULT_TABLE table=test-table [id=ef26e2cf68d24008bd3de4acc27e6a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:07.038466 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8230e09cc08b479d9369002bb38b4bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:07.039362 8512 ts_tablet_manager.cc:1905] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:07.041795 7965 catalog_manager.cc:4841] TS 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881): tablet ca00f74dbf4d48b69e70cd1176271880 (table test-table [id=bd4dd53e699d4ab9be118f7b922e7531]) successfully deleted
I20260927 22:52:07.042910 7967 catalog_manager.cc:4841] TS b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097): tablet ca00f74dbf4d48b69e70cd1176271880 (table test-table [id=bd4dd53e699d4ab9be118f7b922e7531]) successfully deleted
I20260927 22:52:07.046316 8489 tablet_bootstrap.cc:492] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b: Bootstrap starting.
I20260927 22:52:07.046432 8490 tablet_bootstrap.cc:492] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769: Bootstrap starting.
I20260927 22:52:07.046628 8512 ts_tablet_manager.cc:1918] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:07.046828 8512 log.cc:1196] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/ca00f74dbf4d48b69e70cd1176271880
I20260927 22:52:07.047429 8512 ts_tablet_manager.cc:1939] T ca00f74dbf4d48b69e70cd1176271880 P 3e52b061730e43ef9b024497962ea92b: Deleting consensus metadata
I20260927 22:52:07.047945 8490 tablet_bootstrap.cc:654] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:07.048005 8489 tablet_bootstrap.cc:654] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:07.049144 7964 catalog_manager.cc:4841] TS 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797): tablet ca00f74dbf4d48b69e70cd1176271880 (table test-table [id=bd4dd53e699d4ab9be118f7b922e7531]) successfully deleted
I20260927 22:52:07.050037 8491 tablet_bootstrap.cc:492] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f: Bootstrap starting.
I20260927 22:52:07.051973 8491 tablet_bootstrap.cc:654] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:07.063830 8491 tablet_bootstrap.cc:492] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f: No bootstrap required, opened a new log
I20260927 22:52:07.064101 8491 ts_tablet_manager.cc:1397] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260927 22:52:07.065152 8491 raft_consensus.cc:348] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.065562 8491 raft_consensus.cc:374] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:07.065757 8491 raft_consensus.cc:729] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b514f9b69fce46fe956bf2be257d729f, State: Initialized, Role: FOLLOWER
I20260927 22:52:07.066182 8491 consensus_queue.cc:260] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.071009 8491 ts_tablet_manager.cc:1428] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260927 22:52:07.076360 8489 tablet_bootstrap.cc:492] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b: No bootstrap required, opened a new log
I20260927 22:52:07.076422 8490 tablet_bootstrap.cc:492] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769: No bootstrap required, opened a new log
I20260927 22:52:07.076546 8489 ts_tablet_manager.cc:1397] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.012s
I20260927 22:52:07.076547 8490 ts_tablet_manager.cc:1397] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.015s
I20260927 22:52:07.077327 8490 raft_consensus.cc:348] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.077556 8490 raft_consensus.cc:374] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:07.077431 8489 raft_consensus.cc:348] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.077663 8490 raft_consensus.cc:729] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e202f17d16e44d1bf4fff4c32a23769, State: Initialized, Role: FOLLOWER
I20260927 22:52:07.077680 8489 raft_consensus.cc:374] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:07.077836 8489 raft_consensus.cc:729] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e52b061730e43ef9b024497962ea92b, State: Initialized, Role: FOLLOWER
I20260927 22:52:07.077946 8490 consensus_queue.cc:260] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.078094 8489 consensus_queue.cc:260] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.078544 8490 ts_tablet_manager.cc:1428] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:52:07.078660 8489 ts_tablet_manager.cc:1428] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:52:07.140388 8499 raft_consensus.cc:482] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:52:07.140592 8499 raft_consensus.cc:504] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.141271 8499 leader_election.cc:288] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097), 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881)
I20260927 22:52:07.141769 8136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8230e09cc08b479d9369002bb38b4bd8" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b514f9b69fce46fe956bf2be257d729f" is_pre_election: true
I20260927 22:52:07.142059 8136 raft_consensus.cc:2454] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 0.
I20260927 22:52:07.142006 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8230e09cc08b479d9369002bb38b4bd8" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e202f17d16e44d1bf4fff4c32a23769" is_pre_election: true
I20260927 22:52:07.142232 8281 raft_consensus.cc:2454] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 0.
I20260927 22:52:07.142479 8349 leader_election.cc:302] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e52b061730e43ef9b024497962ea92b, b514f9b69fce46fe956bf2be257d729f; no voters:
I20260927 22:52:07.142750 8499 raft_consensus.cc:2781] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:52:07.142853 8499 raft_consensus.cc:482] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:52:07.142946 8499 raft_consensus.cc:3037] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:07.146267 8499 raft_consensus.cc:504] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.147614 8499 leader_election.cc:288] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 election: Requested vote from peers b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097), 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881)
I20260927 22:52:07.147989 8136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8230e09cc08b479d9369002bb38b4bd8" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b514f9b69fce46fe956bf2be257d729f"
I20260927 22:52:07.148167 8136 raft_consensus.cc:3037] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:07.148351 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8230e09cc08b479d9369002bb38b4bd8" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e202f17d16e44d1bf4fff4c32a23769"
I20260927 22:52:07.148512 8281 raft_consensus.cc:3037] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:07.151096 8136 raft_consensus.cc:2454] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 1.
I20260927 22:52:07.151288 8281 raft_consensus.cc:2454] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 1.
I20260927 22:52:07.151476 8349 leader_election.cc:302] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e52b061730e43ef9b024497962ea92b, b514f9b69fce46fe956bf2be257d729f; no voters:
I20260927 22:52:07.151808 8499 raft_consensus.cc:2781] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:07.151974 8499 raft_consensus.cc:686] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 1 LEADER]: Becoming Leader. State: Replica: 3e52b061730e43ef9b024497962ea92b, State: Running, Role: LEADER
I20260927 22:52:07.152223 8499 consensus_queue.cc:237] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.154831 7979 catalog_manager.cc:5495] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e52b061730e43ef9b024497962ea92b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3e52b061730e43ef9b024497962ea92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } health_report { overall_health: HEALTHY } } }
Sep 27 22:52:07 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:52:07.218720 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.203868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.0.97.62:40851)
0927 22:52:07.204134 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:07.204143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:07.204670 (+ 527us) server_negotiation.cc:407] Connection header received
0927 22:52:07.204862 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:07.204866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:07.204924 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:07.205012 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:07.206498 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.207250 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.209096 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.209484 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.209572 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.213352 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:07.213357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:07.213360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:07.213386 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:07.216023 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:07.216804 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:07.216807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:07.216810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:07.216923 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:07.217281 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:07.217284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:07.217286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:07.217553 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:07.217703 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:07.218149 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 22:52:07.218441 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":129}
I20260927 22:52:07.218765 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.203879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40851 (local address 127.0.0.1:39148)
0927 22:52:07.204546 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:07.204567 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:07.204706 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:07.205073 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:07.205079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:07.206019 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:07.206345 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:07.206358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.207415 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.207421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:07.208956 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:07.208970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.209677 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.209684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:07.209852 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.210515 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:07.210543 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:07.213161 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:07.216149 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:07.216162 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:07.216169 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:07.216692 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:07.217026 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:07.217030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:07.217034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:07.217184 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:07.217700 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:07.217708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:07.217902 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 22:52:07.218063 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 22:52:07.218319 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":360,"threads_started":1}
I20260927 22:52:07.223836 7979 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39148:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:52:07.224987 7979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:52:07.232100 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 7504f37975dc4d86aee88d952df4530f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8a205dab6e240159da6115b83f78b62]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:52:07.232458 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 7504f37975dc4d86aee88d952df4530f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8a205dab6e240159da6115b83f78b62]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:52:07.232641 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7504f37975dc4d86aee88d952df4530f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:07.232946 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7504f37975dc4d86aee88d952df4530f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:07.234246 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 7504f37975dc4d86aee88d952df4530f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8a205dab6e240159da6115b83f78b62]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:52:07.234674 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7504f37975dc4d86aee88d952df4530f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:07.239642 8491 tablet_bootstrap.cc:492] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: Bootstrap starting.
I20260927 22:52:07.241062 8491 tablet_bootstrap.cc:654] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:07.247468 8489 tablet_bootstrap.cc:492] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: Bootstrap starting.
I20260927 22:52:07.248967 8489 tablet_bootstrap.cc:654] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:07.250711 8490 tablet_bootstrap.cc:492] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: Bootstrap starting.
I20260927 22:52:07.251541 8489 tablet_bootstrap.cc:492] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: No bootstrap required, opened a new log
I20260927 22:52:07.251719 8489 ts_tablet_manager.cc:1397] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:52:07.252292 8490 tablet_bootstrap.cc:654] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:07.252576 8489 raft_consensus.cc:348] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.252836 8489 raft_consensus.cc:374] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:07.252933 8489 raft_consensus.cc:729] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e52b061730e43ef9b024497962ea92b, State: Initialized, Role: FOLLOWER
I20260927 22:52:07.253214 8489 consensus_queue.cc:260] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.253825 8499 tablet_replica.cc:439] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } }
I20260927 22:52:07.254179 8499 tablet_replica.cc:442] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:52:07.254711 8489 ts_tablet_manager.cc:1428] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:52:07.256664 8490 tablet_bootstrap.cc:492] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: No bootstrap required, opened a new log
I20260927 22:52:07.256843 8490 ts_tablet_manager.cc:1397] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 22:52:07.257695 8491 tablet_bootstrap.cc:492] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: No bootstrap required, opened a new log
I20260927 22:52:07.257560 8490 raft_consensus.cc:348] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.257843 8490 raft_consensus.cc:374] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:07.257933 8490 raft_consensus.cc:729] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e202f17d16e44d1bf4fff4c32a23769, State: Initialized, Role: FOLLOWER
I20260927 22:52:07.257983 8491 ts_tablet_manager.cc:1397] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.009s
I20260927 22:52:07.258176 8490 consensus_queue.cc:260] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.258736 8490 ts_tablet_manager.cc:1428] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:52:07.258603 8496 tablet_replica.cc:439] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } }
I20260927 22:52:07.258741 8491 raft_consensus.cc:348] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.258944 8496 tablet_replica.cc:442] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:52:07.258956 8491 raft_consensus.cc:374] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:07.259021 8491 raft_consensus.cc:729] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b514f9b69fce46fe956bf2be257d729f, State: Initialized, Role: FOLLOWER
I20260927 22:52:07.259255 8491 consensus_queue.cc:260] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.259737 8497 tablet_replica.cc:439] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } }
I20260927 22:52:07.260073 8497 tablet_replica.cc:442] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:52:07.260569 8491 ts_tablet_manager.cc:1428] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 22:52:07.609921 8499 consensus_queue.cc:1035] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:07.614614 8529 consensus_queue.cc:1035] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:07.629863 8496 raft_consensus.cc:482] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:52:07.630039 8496 raft_consensus.cc:504] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.631525 8496 leader_election.cc:288] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097), 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797)
I20260927 22:52:07.636224 8529 raft_consensus.cc:482] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:52:07.636396 8529 raft_consensus.cc:504] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.637003 8529 leader_election.cc:288] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097), 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881)
I20260927 22:52:07.637389 8136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7504f37975dc4d86aee88d952df4530f" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b514f9b69fce46fe956bf2be257d729f" is_pre_election: true
I20260927 22:52:07.637624 8136 raft_consensus.cc:2454] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 0.
I20260927 22:52:07.637955 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7504f37975dc4d86aee88d952df4530f" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e202f17d16e44d1bf4fff4c32a23769" is_pre_election: true
I20260927 22:52:07.638037 8349 leader_election.cc:302] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e52b061730e43ef9b024497962ea92b, b514f9b69fce46fe956bf2be257d729f; no voters:
I20260927 22:52:07.638170 8281 raft_consensus.cc:2454] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 0.
I20260927 22:52:07.638299 8529 raft_consensus.cc:2781] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:52:07.638463 8529 raft_consensus.cc:482] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:52:07.638577 8529 raft_consensus.cc:3037] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:07.638582 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.631744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40797 (local address 127.0.97.2:45859)
0927 22:52:07.632358 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:07.632392 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:07.632526 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:07.633211 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:07.633215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:07.633237 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:07.633506 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:07.633514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.634528 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.634534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:07.636136 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:07.636150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.637781 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.637786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:07.637995 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.638021 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:52:07.638102 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 22:52:07.638223 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":130,"thread_start_us":414,"threads_started":1}
I20260927 22:52:07.638999 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.632050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45859 (local address 127.0.97.3:40797)
0927 22:52:07.632965 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:07.632970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:07.632984 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:52:07.633034 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:07.633038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:07.633087 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:07.633193 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:07.633724 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.634388 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.636280 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.637562 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.638010 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.638119 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:07.638168 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 22:52:07.638278 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":708,"threads_started":1}
I20260927 22:52:07.639273 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.631755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34575 (local address 127.0.97.1:38097)
0927 22:52:07.633008 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:07.633012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:07.633026 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:52:07.633071 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:07.633074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:07.633119 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:07.633238 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:07.633852 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.634531 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.636281 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.637466 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.637622 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.637732 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:07.638895 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0927 22:52:07.639024 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":898,"threads_started":1}
I20260927 22:52:07.639286 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.631685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38097 (local address 127.0.97.2:34575)
0927 22:52:07.632206 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:07.632231 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:07.632409 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:07.633249 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:07.633254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:07.633287 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:07.633709 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:07.633719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.634655 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.634664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:07.636089 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:07.636099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.637687 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.637696 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:07.637896 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.638733 (+ 837us) client_negotiation.cc:769] Sending connection context
0927 22:52:07.638811 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 22:52:07.638941 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":28,"thread_start_us":220,"threads_started":1}
I20260927 22:52:07.639958 8136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7504f37975dc4d86aee88d952df4530f" candidate_uuid: "7e202f17d16e44d1bf4fff4c32a23769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b514f9b69fce46fe956bf2be257d729f" is_pre_election: true
I20260927 22:52:07.639957 8415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7504f37975dc4d86aee88d952df4530f" candidate_uuid: "7e202f17d16e44d1bf4fff4c32a23769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e52b061730e43ef9b024497962ea92b" is_pre_election: true
I20260927 22:52:07.640162 8136 raft_consensus.cc:2454] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e202f17d16e44d1bf4fff4c32a23769 in term 0.
I20260927 22:52:07.640616 8210 leader_election.cc:302] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e202f17d16e44d1bf4fff4c32a23769, b514f9b69fce46fe956bf2be257d729f; no voters:
I20260927 22:52:07.640920 8496 raft_consensus.cc:2781] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:52:07.641019 8496 raft_consensus.cc:482] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:52:07.641153 8496 raft_consensus.cc:3037] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:07.641381 8529 raft_consensus.cc:504] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.641767 8415 raft_consensus.cc:2379] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e202f17d16e44d1bf4fff4c32a23769 in current term 1: Already voted for candidate 3e52b061730e43ef9b024497962ea92b in this term.
I20260927 22:52:07.641976 8529 leader_election.cc:288] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 election: Requested vote from peers b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097), 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881)
I20260927 22:52:07.642339 8136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7504f37975dc4d86aee88d952df4530f" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b514f9b69fce46fe956bf2be257d729f"
I20260927 22:52:07.642534 8136 raft_consensus.cc:3037] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:07.642588 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7504f37975dc4d86aee88d952df4530f" candidate_uuid: "3e52b061730e43ef9b024497962ea92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e202f17d16e44d1bf4fff4c32a23769"
I20260927 22:52:07.644117 8496 raft_consensus.cc:504] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.644615 8281 raft_consensus.cc:2379] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3e52b061730e43ef9b024497962ea92b in current term 1: Already voted for candidate 7e202f17d16e44d1bf4fff4c32a23769 in this term.
I20260927 22:52:07.644687 8496 leader_election.cc:288] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [CANDIDATE]: Term 1 election: Requested vote from peers b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097), 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797)
I20260927 22:52:07.645226 8415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7504f37975dc4d86aee88d952df4530f" candidate_uuid: "7e202f17d16e44d1bf4fff4c32a23769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e52b061730e43ef9b024497962ea92b"
I20260927 22:52:07.645342 8136 raft_consensus.cc:2454] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e52b061730e43ef9b024497962ea92b in term 1.
I20260927 22:52:07.645447 8415 raft_consensus.cc:2379] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e202f17d16e44d1bf4fff4c32a23769 in current term 1: Already voted for candidate 3e52b061730e43ef9b024497962ea92b in this term.
I20260927 22:52:07.645691 8137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7504f37975dc4d86aee88d952df4530f" candidate_uuid: "7e202f17d16e44d1bf4fff4c32a23769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b514f9b69fce46fe956bf2be257d729f"
I20260927 22:52:07.645857 8349 leader_election.cc:302] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e52b061730e43ef9b024497962ea92b, b514f9b69fce46fe956bf2be257d729f; no voters: 7e202f17d16e44d1bf4fff4c32a23769
I20260927 22:52:07.645951 8137 raft_consensus.cc:2379] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e202f17d16e44d1bf4fff4c32a23769 in current term 1: Already voted for candidate 3e52b061730e43ef9b024497962ea92b in this term.
I20260927 22:52:07.646173 8529 raft_consensus.cc:2781] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:07.646354 8529 raft_consensus.cc:686] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [term 1 LEADER]: Becoming Leader. State: Replica: 3e52b061730e43ef9b024497962ea92b, State: Running, Role: LEADER
I20260927 22:52:07.646322 8210 leader_election.cc:302] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e202f17d16e44d1bf4fff4c32a23769; no voters: 3e52b061730e43ef9b024497962ea92b, b514f9b69fce46fe956bf2be257d729f
I20260927 22:52:07.646598 8529 consensus_queue.cc:237] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.646728 8496 raft_consensus.cc:2726] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 22:52:07.648094 8495 tablet_replica.cc:439] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: TxnStatusTablet state changed. Reason: New leader 3e52b061730e43ef9b024497962ea92b. Latest consensus state: current_term: 1 leader_uuid: "3e52b061730e43ef9b024497962ea92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } }
I20260927 22:52:07.648360 8495 tablet_replica.cc:442] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:52:07.648887 8538 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:52:07.649078 8538 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:52:07.649297 7979 catalog_manager.cc:5495] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e52b061730e43ef9b024497962ea92b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3e52b061730e43ef9b024497962ea92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } health_report { overall_health: HEALTHY } } }
I20260927 22:52:07.710762 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.705451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.0.97.3:40797)
0927 22:52:07.705671 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:07.705677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:07.705698 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:52:07.705923 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:07.705928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:07.706038 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:07.706135 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:07.706438 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.707564 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.708869 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.709516 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.709689 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.710063 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:07.710216 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:07.710302 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:07.710422 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 22:52:07.710495 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:52:07.710788 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.705362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40797 (local address 127.0.0.1:60398)
0927 22:52:07.705579 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:07.705635 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:07.705749 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:07.706160 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:07.706163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:07.706172 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:52:07.706313 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:07.706319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.707698 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.707703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:07.708715 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:07.708726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.709707 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.709711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:07.709842 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.709892 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:52:07.710307 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:52:07.710319 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:52:07.710381 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 22:52:07.710485 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:52:07.717494 8281 raft_consensus.cc:1261] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Refusing update from remote peer 3e52b061730e43ef9b024497962ea92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:07.717662 8137 raft_consensus.cc:1261] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Refusing update from remote peer 3e52b061730e43ef9b024497962ea92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:07.718153 8529 consensus_queue.cc:1035] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:07.718429 8499 consensus_queue.cc:1035] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:07.728772 8496 tablet_replica.cc:439] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: TxnStatusTablet state changed. Reason: New leader 3e52b061730e43ef9b024497962ea92b. Latest consensus state: current_term: 1 leader_uuid: "3e52b061730e43ef9b024497962ea92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } }
I20260927 22:52:07.729090 8496 tablet_replica.cc:442] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:52:07.742856 8496 tablet_replica.cc:439] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e52b061730e43ef9b024497962ea92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } }
I20260927 22:52:07.743100 8496 tablet_replica.cc:442] T 7504f37975dc4d86aee88d952df4530f P 7e202f17d16e44d1bf4fff4c32a23769: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:52:07.745211 8499 tablet_replica.cc:439] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e52b061730e43ef9b024497962ea92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } }
I20260927 22:52:07.745491 8499 tablet_replica.cc:442] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:52:07.747370 8497 tablet_replica.cc:439] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: TxnStatusTablet state changed. Reason: New leader 3e52b061730e43ef9b024497962ea92b. Latest consensus state: current_term: 1 leader_uuid: "3e52b061730e43ef9b024497962ea92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } }
I20260927 22:52:07.747660 8497 tablet_replica.cc:442] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:52:07.751091 8529 tablet_replica.cc:439] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e52b061730e43ef9b024497962ea92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } }
I20260927 22:52:07.751370 8529 tablet_replica.cc:442] T 7504f37975dc4d86aee88d952df4530f P 3e52b061730e43ef9b024497962ea92b: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:52:07.774542 8497 tablet_replica.cc:439] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e52b061730e43ef9b024497962ea92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } } }
I20260927 22:52:07.774789 8497 tablet_replica.cc:442] T 7504f37975dc4d86aee88d952df4530f P b514f9b69fce46fe956bf2be257d729f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:52:07.782377 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.776886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40797 (local address 127.0.97.3:58147)
0927 22:52:07.777505 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:07.777519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:07.777694 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:07.778015 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:07.778019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:07.778029 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:52:07.778283 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:07.778294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.779313 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.779321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:07.780496 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:07.780506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.781204 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.781209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:07.781339 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.781382 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:52:07.781806 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:52:07.781816 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:52:07.781910 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:52:07.782035 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":319,"threads_started":1}
I20260927 22:52:07.782439 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.776907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58147 (local address 127.0.97.3:40797)
0927 22:52:07.777138 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:07.777146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:07.777615 (+ 469us) server_negotiation.cc:407] Connection header received
0927 22:52:07.777847 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:07.777850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:07.777904 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:07.778005 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:07.778434 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.779190 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.780636 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.781072 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.781198 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.781571 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:07.781716 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:07.781806 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:07.781951 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 22:52:07.782035 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:52:07.849102 7979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:07.849319 7979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:07.851951 7979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9: Sending DeleteTablet for 3 replicas of tablet 8230e09cc08b479d9369002bb38b4bd8
I20260927 22:52:07.852578 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet 8230e09cc08b479d9369002bb38b4bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51582
I20260927 22:52:07.852685 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet 8230e09cc08b479d9369002bb38b4bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49084
I20260927 22:52:07.852736 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet 8230e09cc08b479d9369002bb38b4bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60392
I20260927 22:52:07.862121 8551 tablet_replica.cc:330] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769: stopping tablet replica
I20260927 22:52:07.862303 8550 tablet_replica.cc:330] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f: stopping tablet replica
I20260927 22:52:07.862329 8551 raft_consensus.cc:2229] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:07.862491 8550 raft_consensus.cc:2229] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:07.862584 8551 raft_consensus.cc:2258] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:07.862741 8550 raft_consensus.cc:2258] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:07.863186 8552 tablet_replica.cc:330] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b: stopping tablet replica
I20260927 22:52:07.863399 8552 raft_consensus.cc:2229] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:52:07.863821 8550 ts_tablet_manager.cc:1905] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:07.863803 8552 raft_consensus.cc:2258] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:07.864866 8551 ts_tablet_manager.cc:1905] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:07.866408 8552 ts_tablet_manager.cc:1905] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:07.870400 8550 ts_tablet_manager.cc:1918] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:52:07.870529 8550 log.cc:1196] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/8230e09cc08b479d9369002bb38b4bd8
I20260927 22:52:07.870971 8550 ts_tablet_manager.cc:1939] T 8230e09cc08b479d9369002bb38b4bd8 P b514f9b69fce46fe956bf2be257d729f: Deleting consensus metadata
I20260927 22:52:07.871909 7967 catalog_manager.cc:4841] TS b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097): tablet 8230e09cc08b479d9369002bb38b4bd8 (table test-table [id=ef26e2cf68d24008bd3de4acc27e6a96]) successfully deleted
I20260927 22:52:07.875952 8551 ts_tablet_manager.cc:1918] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:52:07.876111 8551 log.cc:1196] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/8230e09cc08b479d9369002bb38b4bd8
I20260927 22:52:07.876582 8551 ts_tablet_manager.cc:1939] T 8230e09cc08b479d9369002bb38b4bd8 P 7e202f17d16e44d1bf4fff4c32a23769: Deleting consensus metadata
I20260927 22:52:07.877063 8552 ts_tablet_manager.cc:1918] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:52:07.877198 8552 log.cc:1196] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/8230e09cc08b479d9369002bb38b4bd8
I20260927 22:52:07.877656 8552 ts_tablet_manager.cc:1939] T 8230e09cc08b479d9369002bb38b4bd8 P 3e52b061730e43ef9b024497962ea92b: Deleting consensus metadata
I20260927 22:52:07.878556 7964 catalog_manager.cc:4841] TS 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797): tablet 8230e09cc08b479d9369002bb38b4bd8 (table test-table [id=ef26e2cf68d24008bd3de4acc27e6a96]) successfully deleted
I20260927 22:52:07.881419 7965 catalog_manager.cc:4841] TS 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881): tablet 8230e09cc08b479d9369002bb38b4bd8 (table test-table [id=ef26e2cf68d24008bd3de4acc27e6a96]) successfully deleted
Sep 27 22:52:07 dist-test-slave-df53 krb5kdc[7935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:52:07 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:07 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:52:07.931568 8560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:07.903524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.0.97.62:40851)
0927 22:52:07.906086 (+ 2562us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:07.906093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:07.906381 (+ 288us) server_negotiation.cc:407] Connection header received
0927 22:52:07.906545 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:07.906551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:07.906635 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:07.906750 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:07.908143 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.909285 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.910580 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.911205 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.911338 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.931238 (+ 19900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39150: BlockingRecv error: recv got EOF from 127.0.0.1:39150 (error 108)
Metrics: {"server-negotiator.queue_time_us":2409,"thread_start_us":2241,"threads_started":1}
Sep 27 22:52:07 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 22:52:07.949321 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:07.934226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.0.97.62:40851)
0927 22:52:07.934442 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:07.934448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:07.934689 (+ 241us) server_negotiation.cc:407] Connection header received
0927 22:52:07.934872 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:07.934877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:07.934952 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:07.935034 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:07.936507 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.937176 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.938979 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:07.939394 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:07.939487 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:07.943695 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:07.943700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:07.943703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:07.943734 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:07.946775 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:07.947536 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:07.947540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:07.947543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:07.947670 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:07.948072 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:07.948075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:07.948077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:07.948315 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:07.948493 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:07.948835 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 22:52:07.949026 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:52:07.952733 7979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:52:07.953313 7979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:52:07.960117 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 09dd109e8e424de9b526dc8f26eefb5c (DEFAULT_TABLE table=test-table [id=b453e3af43bf4ccaae886d92964c9c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:07.960541 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09dd109e8e424de9b526dc8f26eefb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:07.960454 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 09dd109e8e424de9b526dc8f26eefb5c (DEFAULT_TABLE table=test-table [id=b453e3af43bf4ccaae886d92964c9c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:07.960541 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 09dd109e8e424de9b526dc8f26eefb5c (DEFAULT_TABLE table=test-table [id=b453e3af43bf4ccaae886d92964c9c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:07.960897 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09dd109e8e424de9b526dc8f26eefb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:07.960917 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09dd109e8e424de9b526dc8f26eefb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:07.973528 8568 tablet_bootstrap.cc:492] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b: Bootstrap starting.
I20260927 22:52:07.975835 8568 tablet_bootstrap.cc:654] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:07.977313 8566 tablet_bootstrap.cc:492] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f: Bootstrap starting.
I20260927 22:52:07.977691 8567 tablet_bootstrap.cc:492] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769: Bootstrap starting.
I20260927 22:52:07.979205 8566 tablet_bootstrap.cc:654] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:07.979938 8567 tablet_bootstrap.cc:654] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:07.981648 8568 tablet_bootstrap.cc:492] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b: No bootstrap required, opened a new log
I20260927 22:52:07.981925 8568 ts_tablet_manager.cc:1397] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20260927 22:52:07.982714 8566 tablet_bootstrap.cc:492] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f: No bootstrap required, opened a new log
I20260927 22:52:07.982860 8566 ts_tablet_manager.cc:1397] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 22:52:07.983610 8566 raft_consensus.cc:348] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.983879 8566 raft_consensus.cc:374] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:07.983968 8566 raft_consensus.cc:729] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b514f9b69fce46fe956bf2be257d729f, State: Initialized, Role: FOLLOWER
I20260927 22:52:07.984216 8566 consensus_queue.cc:260] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.983404 8568 raft_consensus.cc:348] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.984771 8568 raft_consensus.cc:374] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:07.984982 8568 raft_consensus.cc:729] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e52b061730e43ef9b024497962ea92b, State: Initialized, Role: FOLLOWER
I20260927 22:52:07.985357 8568 consensus_queue.cc:260] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.987099 8566 ts_tablet_manager.cc:1428] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:52:07.988433 8567 tablet_bootstrap.cc:492] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769: No bootstrap required, opened a new log
I20260927 22:52:07.988572 8567 ts_tablet_manager.cc:1397] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260927 22:52:07.988785 8568 ts_tablet_manager.cc:1428] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260927 22:52:07.989356 8567 raft_consensus.cc:348] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.989604 8567 raft_consensus.cc:374] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:07.989715 8567 raft_consensus.cc:729] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e202f17d16e44d1bf4fff4c32a23769, State: Initialized, Role: FOLLOWER
I20260927 22:52:07.989964 8567 consensus_queue.cc:260] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:07.991075 8567 ts_tablet_manager.cc:1428] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:52:08.064744 8497 raft_consensus.cc:482] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:52:08.065053 8497 raft_consensus.cc:504] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.066620 8497 leader_election.cc:288] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881), 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797)
I20260927 22:52:08.075179 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.067024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40797 (local address 127.0.97.1:50139)
0927 22:52:08.068552 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:08.068569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:08.068705 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:08.069178 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:08.069182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:08.069206 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:08.069629 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:08.069641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.071078 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.071083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:08.072580 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:08.072591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.074292 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.074300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:08.074575 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.074612 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:52:08.074707 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:52:08.074829 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1418,"thread_start_us":1267,"threads_started":1}
I20260927 22:52:08.075204 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.067045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50139 (local address 127.0.97.3:40797)
0927 22:52:08.067450 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:08.067457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:08.068680 (+ 1223us) server_negotiation.cc:407] Connection header received
0927 22:52:08.068864 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:08.068869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:08.068935 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:08.068993 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:08.069815 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.070895 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.072730 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.074083 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.074212 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.074314 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:08.074748 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 22:52:08.074930 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260927 22:52:08.075453 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.066796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39881 (local address 127.0.97.1:37337)
0927 22:52:08.067596 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:08.067620 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:08.067785 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:08.068785 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:08.068789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:08.068816 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:08.069101 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:08.069109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.070596 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.070601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:08.072801 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:08.072815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.074485 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.074492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:08.074661 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.074715 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 22:52:08.074803 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:52:08.074928 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":662,"threads_started":1}
I20260927 22:52:08.075914 8415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09dd109e8e424de9b526dc8f26eefb5c" candidate_uuid: "b514f9b69fce46fe956bf2be257d729f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e52b061730e43ef9b024497962ea92b" is_pre_election: true
I20260927 22:52:08.076083 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.067162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37337 (local address 127.0.97.2:39881)
0927 22:52:08.068411 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:08.068417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:08.068439 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:52:08.068507 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:08.068515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:08.068593 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:08.068766 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:08.069277 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.070467 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.072958 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.074247 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.074364 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.074546 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:08.075633 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0927 22:52:08.075772 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":926,"threads_started":1}
I20260927 22:52:08.076203 8415 raft_consensus.cc:2454] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b514f9b69fce46fe956bf2be257d729f in term 0.
I20260927 22:52:08.076567 8280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09dd109e8e424de9b526dc8f26eefb5c" candidate_uuid: "b514f9b69fce46fe956bf2be257d729f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e202f17d16e44d1bf4fff4c32a23769" is_pre_election: true
I20260927 22:52:08.076674 8067 leader_election.cc:302] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e52b061730e43ef9b024497962ea92b, b514f9b69fce46fe956bf2be257d729f; no voters:
I20260927 22:52:08.076836 8280 raft_consensus.cc:2454] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b514f9b69fce46fe956bf2be257d729f in term 0.
I20260927 22:52:08.077172 8497 raft_consensus.cc:2781] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:52:08.077292 8497 raft_consensus.cc:482] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:52:08.077390 8497 raft_consensus.cc:3037] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:08.079463 8497 raft_consensus.cc:504] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.080061 8497 leader_election.cc:288] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [CANDIDATE]: Term 1 election: Requested vote from peers 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881), 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797)
I20260927 22:52:08.080428 8280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09dd109e8e424de9b526dc8f26eefb5c" candidate_uuid: "b514f9b69fce46fe956bf2be257d729f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e202f17d16e44d1bf4fff4c32a23769"
I20260927 22:52:08.080513 8415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09dd109e8e424de9b526dc8f26eefb5c" candidate_uuid: "b514f9b69fce46fe956bf2be257d729f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e52b061730e43ef9b024497962ea92b"
I20260927 22:52:08.080621 8280 raft_consensus.cc:3037] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:08.080724 8415 raft_consensus.cc:3037] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:08.083166 8415 raft_consensus.cc:2454] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b514f9b69fce46fe956bf2be257d729f in term 1.
I20260927 22:52:08.083503 8067 leader_election.cc:302] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e52b061730e43ef9b024497962ea92b, b514f9b69fce46fe956bf2be257d729f; no voters:
I20260927 22:52:08.083503 8280 raft_consensus.cc:2454] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b514f9b69fce46fe956bf2be257d729f in term 1.
I20260927 22:52:08.083779 8497 raft_consensus.cc:2781] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:08.084249 8497 raft_consensus.cc:686] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 1 LEADER]: Becoming Leader. State: Replica: b514f9b69fce46fe956bf2be257d729f, State: Running, Role: LEADER
I20260927 22:52:08.084681 8497 consensus_queue.cc:237] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.088018 7979 catalog_manager.cc:5495] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f reported cstate change: term changed from 0 to 1, leader changed from <none> to b514f9b69fce46fe956bf2be257d729f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b514f9b69fce46fe956bf2be257d729f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } health_report { overall_health: UNKNOWN } } }
I20260927 22:52:08.128906 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.123620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.0.97.1:38097)
0927 22:52:08.123829 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:08.123835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:08.123942 (+ 107us) server_negotiation.cc:407] Connection header received
0927 22:52:08.124224 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:08.124228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:08.124323 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:08.124496 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:08.124838 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.125546 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.126872 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.127280 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.127422 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.127819 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:08.128343 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:08.128493 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:08.128591 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 22:52:08.128665 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:52:08.132350 8280 raft_consensus.cc:1261] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Refusing update from remote peer b514f9b69fce46fe956bf2be257d729f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:08.132714 8415 raft_consensus.cc:1261] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Refusing update from remote peer b514f9b69fce46fe956bf2be257d729f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:08.133031 8575 consensus_queue.cc:1035] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:08.133383 8497 consensus_queue.cc:1035] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:08.159879 7980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:08.160096 7980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:08.162915 7980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9: Sending DeleteTablet for 3 replicas of tablet 09dd109e8e424de9b526dc8f26eefb5c
I20260927 22:52:08.163604 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet 09dd109e8e424de9b526dc8f26eefb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51582
I20260927 22:52:08.163882 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet 09dd109e8e424de9b526dc8f26eefb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60392
I20260927 22:52:08.163902 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet 09dd109e8e424de9b526dc8f26eefb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49084
I20260927 22:52:08.164227 8550 tablet_replica.cc:330] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f: stopping tablet replica
I20260927 22:52:08.164466 8550 raft_consensus.cc:2229] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:52:08.164932 8550 raft_consensus.cc:2258] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:08.165234 8552 tablet_replica.cc:330] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b: stopping tablet replica
I20260927 22:52:08.165166 7980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:52:08.165421 8552 raft_consensus.cc:2229] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:08.165735 8551 tablet_replica.cc:330] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769: stopping tablet replica
I20260927 22:52:08.165728 8552 raft_consensus.cc:2258] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 22:52:08.165817 7980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:52:08.165915 8551 raft_consensus.cc:2229] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:08.166131 8551 raft_consensus.cc:2258] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:08.166800 8552 ts_tablet_manager.cc:1905] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:08.167148 8551 ts_tablet_manager.cc:1905] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:08.171831 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 55b282e0644542b6b03cb3932e9e097c (DEFAULT_TABLE table=test-table [id=686ae743bd6c4cf7a37c496f3c22c64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:08.172252 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b282e0644542b6b03cb3932e9e097c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:08.172365 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 55b282e0644542b6b03cb3932e9e097c (DEFAULT_TABLE table=test-table [id=686ae743bd6c4cf7a37c496f3c22c64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:08.172729 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b282e0644542b6b03cb3932e9e097c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:08.173215 8551 ts_tablet_manager.cc:1918] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:08.173352 8551 log.cc:1196] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/09dd109e8e424de9b526dc8f26eefb5c
I20260927 22:52:08.173787 8551 ts_tablet_manager.cc:1939] T 09dd109e8e424de9b526dc8f26eefb5c P 7e202f17d16e44d1bf4fff4c32a23769: Deleting consensus metadata
I20260927 22:52:08.176057 8259 tablet_service.cc:1467] Processing CreateTablet for tablet 55b282e0644542b6b03cb3932e9e097c (DEFAULT_TABLE table=test-table [id=686ae743bd6c4cf7a37c496f3c22c64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:08.176460 8259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b282e0644542b6b03cb3932e9e097c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:08.178476 8550 ts_tablet_manager.cc:1905] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:08.179579 7965 catalog_manager.cc:4841] TS 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881): tablet 09dd109e8e424de9b526dc8f26eefb5c (table test-table [id=b453e3af43bf4ccaae886d92964c9c85]) successfully deleted
I20260927 22:52:08.181797 8568 tablet_bootstrap.cc:492] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b: Bootstrap starting.
I20260927 22:52:08.183261 8568 tablet_bootstrap.cc:654] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:08.184994 8550 ts_tablet_manager.cc:1918] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:08.185150 8550 log.cc:1196] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/09dd109e8e424de9b526dc8f26eefb5c
I20260927 22:52:08.185550 8550 ts_tablet_manager.cc:1939] T 09dd109e8e424de9b526dc8f26eefb5c P b514f9b69fce46fe956bf2be257d729f: Deleting consensus metadata
I20260927 22:52:08.186630 7967 catalog_manager.cc:4841] TS b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097): tablet 09dd109e8e424de9b526dc8f26eefb5c (table test-table [id=b453e3af43bf4ccaae886d92964c9c85]) successfully deleted
I20260927 22:52:08.188298 8567 tablet_bootstrap.cc:492] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769: Bootstrap starting.
I20260927 22:52:08.189920 8567 tablet_bootstrap.cc:654] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:08.191315 8566 tablet_bootstrap.cc:492] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f: Bootstrap starting.
I20260927 22:52:08.192649 8568 tablet_bootstrap.cc:492] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b: No bootstrap required, opened a new log
I20260927 22:52:08.192785 8568 ts_tablet_manager.cc:1397] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20260927 22:52:08.192994 8566 tablet_bootstrap.cc:654] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:08.193286 8567 tablet_bootstrap.cc:492] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769: No bootstrap required, opened a new log
I20260927 22:52:08.193498 8567 ts_tablet_manager.cc:1397] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 22:52:08.193569 8568 raft_consensus.cc:348] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.193840 8568 raft_consensus.cc:374] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:08.193934 8568 raft_consensus.cc:729] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e52b061730e43ef9b024497962ea92b, State: Initialized, Role: FOLLOWER
I20260927 22:52:08.194294 8568 consensus_queue.cc:260] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.194845 8567 raft_consensus.cc:348] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.195138 8567 raft_consensus.cc:374] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:08.195255 8567 raft_consensus.cc:729] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e202f17d16e44d1bf4fff4c32a23769, State: Initialized, Role: FOLLOWER
I20260927 22:52:08.195552 8567 consensus_queue.cc:260] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.197867 8568 ts_tablet_manager.cc:1428] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260927 22:52:08.199244 8567 ts_tablet_manager.cc:1428] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260927 22:52:08.202642 8552 ts_tablet_manager.cc:1918] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:08.202785 8552 log.cc:1196] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/09dd109e8e424de9b526dc8f26eefb5c
I20260927 22:52:08.202877 8566 tablet_bootstrap.cc:492] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f: No bootstrap required, opened a new log
I20260927 22:52:08.203030 8566 ts_tablet_manager.cc:1397] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260927 22:52:08.203269 8552 ts_tablet_manager.cc:1939] T 09dd109e8e424de9b526dc8f26eefb5c P 3e52b061730e43ef9b024497962ea92b: Deleting consensus metadata
I20260927 22:52:08.203804 8566 raft_consensus.cc:348] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.204253 8566 raft_consensus.cc:374] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:08.204229 7964 catalog_manager.cc:4841] TS 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797): tablet 09dd109e8e424de9b526dc8f26eefb5c (table test-table [id=b453e3af43bf4ccaae886d92964c9c85]) successfully deleted
I20260927 22:52:08.204449 8566 raft_consensus.cc:729] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b514f9b69fce46fe956bf2be257d729f, State: Initialized, Role: FOLLOWER
I20260927 22:52:08.204728 8566 consensus_queue.cc:260] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.205220 8566 ts_tablet_manager.cc:1428] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 22:52:08.230966 8580 raft_consensus.cc:482] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:52:08.231227 8580 raft_consensus.cc:504] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.232079 8580 leader_election.cc:288] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881), 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797)
I20260927 22:52:08.232497 8280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b282e0644542b6b03cb3932e9e097c" candidate_uuid: "b514f9b69fce46fe956bf2be257d729f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e202f17d16e44d1bf4fff4c32a23769" is_pre_election: true
I20260927 22:52:08.232715 8280 raft_consensus.cc:2454] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b514f9b69fce46fe956bf2be257d729f in term 0.
I20260927 22:52:08.232689 8415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b282e0644542b6b03cb3932e9e097c" candidate_uuid: "b514f9b69fce46fe956bf2be257d729f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e52b061730e43ef9b024497962ea92b" is_pre_election: true
I20260927 22:52:08.232899 8415 raft_consensus.cc:2454] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b514f9b69fce46fe956bf2be257d729f in term 0.
I20260927 22:52:08.233323 8067 leader_election.cc:302] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3e52b061730e43ef9b024497962ea92b, 7e202f17d16e44d1bf4fff4c32a23769, b514f9b69fce46fe956bf2be257d729f; no voters:
I20260927 22:52:08.233615 8580 raft_consensus.cc:2781] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:52:08.233733 8580 raft_consensus.cc:482] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:52:08.233801 8580 raft_consensus.cc:3037] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:08.236323 8580 raft_consensus.cc:504] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.237182 8280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b282e0644542b6b03cb3932e9e097c" candidate_uuid: "b514f9b69fce46fe956bf2be257d729f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e202f17d16e44d1bf4fff4c32a23769"
I20260927 22:52:08.237234 8415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b282e0644542b6b03cb3932e9e097c" candidate_uuid: "b514f9b69fce46fe956bf2be257d729f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e52b061730e43ef9b024497962ea92b"
I20260927 22:52:08.237389 8280 raft_consensus.cc:3037] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:08.237393 8415 raft_consensus.cc:3037] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:08.240233 8415 raft_consensus.cc:2454] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b514f9b69fce46fe956bf2be257d729f in term 1.
I20260927 22:52:08.240238 8280 raft_consensus.cc:2454] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b514f9b69fce46fe956bf2be257d729f in term 1.
I20260927 22:52:08.240599 8067 leader_election.cc:302] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e52b061730e43ef9b024497962ea92b, b514f9b69fce46fe956bf2be257d729f; no voters:
I20260927 22:52:08.240896 8575 raft_consensus.cc:2781] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:08.241135 8575 raft_consensus.cc:686] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 1 LEADER]: Becoming Leader. State: Replica: b514f9b69fce46fe956bf2be257d729f, State: Running, Role: LEADER
I20260927 22:52:08.241390 8575 consensus_queue.cc:237] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } }
I20260927 22:52:08.242484 8580 leader_election.cc:288] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [CANDIDATE]: Term 1 election: Requested vote from peers 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881), 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797)
I20260927 22:52:08.244143 7979 catalog_manager.cc:5495] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f reported cstate change: term changed from 0 to 1, leader changed from <none> to b514f9b69fce46fe956bf2be257d729f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b514f9b69fce46fe956bf2be257d729f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b514f9b69fce46fe956bf2be257d729f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 } health_report { overall_health: UNKNOWN } } }
I20260927 22:52:08.272869 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.267074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36315 (local address 127.0.97.3:40797)
0927 22:52:08.267217 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:08.267221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:08.267895 (+ 674us) server_negotiation.cc:407] Connection header received
0927 22:52:08.268104 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:08.268109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:08.268181 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:08.268270 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:08.268802 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.269502 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.270985 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.271575 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.271708 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.272138 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:08.272256 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:08.272349 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:08.272487 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 22:52:08.272575 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:52:08.272989 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.267127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40797 (local address 127.0.97.1:36315)
0927 22:52:08.267798 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:08.267814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:08.267979 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:08.268298 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:08.268303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:08.268314 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:52:08.268659 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:08.268667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.269668 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.269673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:08.270849 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:08.270861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.271748 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.271767 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:08.271902 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.271948 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:52:08.272354 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:52:08.272366 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:52:08.272456 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:52:08.272609 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":324,"threads_started":1}
I20260927 22:52:08.280192 8280 raft_consensus.cc:1261] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Refusing update from remote peer b514f9b69fce46fe956bf2be257d729f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:08.280290 8415 raft_consensus.cc:1261] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Refusing update from remote peer b514f9b69fce46fe956bf2be257d729f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:08.280772 8580 consensus_queue.cc:1035] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e52b061730e43ef9b024497962ea92b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:08.281098 8575 consensus_queue.cc:1035] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e202f17d16e44d1bf4fff4c32a23769" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:08.312409 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.306198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38097 (local address 127.0.97.3:58473)
0927 22:52:08.307134 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:08.307153 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:08.307277 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:08.307632 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:08.307637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:08.307650 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:52:08.307877 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:08.307900 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.308913 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.308920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:08.310520 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:08.310534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.311450 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.311454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:08.311573 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.311614 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:52:08.311963 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:52:08.311970 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:52:08.312028 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 22:52:08.312116 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":670,"threads_started":1}
I20260927 22:52:08.312423 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.306223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58473 (local address 127.0.97.1:38097)
0927 22:52:08.306394 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:08.306402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:08.307222 (+ 820us) server_negotiation.cc:407] Connection header received
0927 22:52:08.307407 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:08.307414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:08.307508 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:08.307585 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:08.308030 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.308772 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.310658 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.311322 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.311450 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.311748 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:08.311868 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:08.311960 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:08.312063 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 22:52:08.312142 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:52:08.338683 7979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:08.338915 7979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:08.341717 7979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b165fd345adb45f58a4879382716d1c9: Sending DeleteTablet for 3 replicas of tablet 55b282e0644542b6b03cb3932e9e097c
I20260927 22:52:08.342389 8259 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b282e0644542b6b03cb3932e9e097c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49084
I20260927 22:52:08.342588 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b282e0644542b6b03cb3932e9e097c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60392
I20260927 22:52:08.342657 8551 tablet_replica.cc:330] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769: stopping tablet replica
I20260927 22:52:08.342813 8551 raft_consensus.cc:2229] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:08.342988 8551 raft_consensus.cc:2258] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:08.343184 8552 tablet_replica.cc:330] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b: stopping tablet replica
I20260927 22:52:08.343358 8552 raft_consensus.cc:2229] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:08.343442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 8051
I20260927 22:52:08.343538 8552 raft_consensus.cc:2258] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:08.344098 8551 ts_tablet_manager.cc:1905] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:08.344130 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b282e0644542b6b03cb3932e9e097c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51582
I20260927 22:52:08.344379 8550 tablet_replica.cc:330] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f: stopping tablet replica
I20260927 22:52:08.344547 8550 raft_consensus.cc:2229] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:52:08.344846 8550 raft_consensus.cc:2258] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:08.346074 8550 ts_tablet_manager.cc:1905] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:08.350397 8552 ts_tablet_manager.cc:1905] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:08.352203 8551 ts_tablet_manager.cc:1918] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:52:08.352347 8551 log.cc:1196] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/55b282e0644542b6b03cb3932e9e097c
I20260927 22:52:08.352844 8551 ts_tablet_manager.cc:1939] T 55b282e0644542b6b03cb3932e9e097c P 7e202f17d16e44d1bf4fff4c32a23769: Deleting consensus metadata
I20260927 22:52:08.354107 7965 catalog_manager.cc:4841] TS 7e202f17d16e44d1bf4fff4c32a23769 (127.0.97.2:39881): tablet 55b282e0644542b6b03cb3932e9e097c (table test-table [id=686ae743bd6c4cf7a37c496f3c22c64b]) successfully deleted
I20260927 22:52:08.357565 8550 ts_tablet_manager.cc:1918] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:52:08.357704 8550 log.cc:1196] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/55b282e0644542b6b03cb3932e9e097c
I20260927 22:52:08.358110 8550 ts_tablet_manager.cc:1939] T 55b282e0644542b6b03cb3932e9e097c P b514f9b69fce46fe956bf2be257d729f: Deleting consensus metadata
I20260927 22:52:08.359109 7967 catalog_manager.cc:4841] TS b514f9b69fce46fe956bf2be257d729f (127.0.97.1:38097): tablet 55b282e0644542b6b03cb3932e9e097c (table test-table [id=686ae743bd6c4cf7a37c496f3c22c64b]) successfully deleted
I20260927 22:52:08.361466 8552 ts_tablet_manager.cc:1918] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:52:08.361622 8552 log.cc:1196] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/55b282e0644542b6b03cb3932e9e097c
I20260927 22:52:08.362021 8552 ts_tablet_manager.cc:1939] T 55b282e0644542b6b03cb3932e9e097c P 3e52b061730e43ef9b024497962ea92b: Deleting consensus metadata
Sep 27 22:52:08 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 22:52:08.362979 7964 catalog_manager.cc:4841] TS 3e52b061730e43ef9b024497962ea92b (127.0.97.3:40797): tablet 55b282e0644542b6b03cb3932e9e097c (table test-table [id=686ae743bd6c4cf7a37c496f3c22c64b]) successfully deleted
I20260927 22:52:08.368978 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.345088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.0.97.1:38097)
0927 22:52:08.346436 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:08.346442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:08.354985 (+ 8543us) server_negotiation.cc:407] Connection header received
0927 22:52:08.355181 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:08.355190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:08.355261 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:08.355369 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:08.356471 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.357233 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.358579 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.359123 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.359295 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.363385 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:08.363396 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:08.363410 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:08.363444 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:08.366404 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:08.367046 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:08.367053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:08.367056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:08.367174 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:08.367536 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:08.367541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:08.367549 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:08.367905 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:08.368239 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:08.368471 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 22:52:08.368667 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228}
I20260927 22:52:08.369639 8182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:08.488780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 8051
I20260927 22:52:08.507953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 8191
Sep 27 22:52:08 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 22:52:08.521757 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.508717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.0.97.2:39881)
0927 22:52:08.508902 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:08.508906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:08.508924 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:52:08.509110 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:08.509114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:08.509179 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:08.509265 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:08.510502 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.511192 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.512327 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.512696 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.512783 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.516548 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:08.516559 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:08.516574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:08.516603 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:08.519346 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:08.519915 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:08.519921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:08.519923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:08.520062 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:08.520385 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:08.520389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:08.520392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:08.520757 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:08.520983 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:08.521216 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:52:08.521374 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:52:08.522418 8321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:08.618142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 8191
I20260927 22:52:08.635744 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 8330
Sep 27 22:52:08 dist-test-slave-df53 krb5kdc[7935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 22:52:08.650297 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:08.636585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.0.97.3:40797)
0927 22:52:08.636761 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:08.636767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:08.636787 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:52:08.636864 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:08.636876 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:08.636942 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:08.637067 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:08.638338 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.639055 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.640302 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:08.640726 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:08.640863 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:08.644992 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:08.645013 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:08.645034 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:08.645066 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:08.647929 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:08.648503 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:08.648512 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:08.648514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:08.648632 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:08.648995 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:08.649000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:08.649002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:08.649332 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:08.649553 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:08.649872 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:52:08.650054 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:52:08.651031 8460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:08.791003 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 8330
I20260927 22:52:08.815214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 7948
I20260927 22:52:08.816381 8020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:08.929394 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 7948
2026-09-27T22:52:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5494 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:09 dist-test-slave-df53 krb5kdc[8600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:09 dist-test-slave-df53 krb5kdc[8600](info): set up 2 sockets
Sep 27 22:52:09 dist-test-slave-df53 krb5kdc[8600](info): commencing operation
krb5kdc: starting...
W20260927 22:52:10.014478 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:10 dist-test-slave-df53 krb5kdc[8600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:10.212805 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42671
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:10.522387 8613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:10.522662 8613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:10.522751 8613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:10.529781 8613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:10.529877 8613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:10.529922 8613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:10.529970 8613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:10.530017 8613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:10.539680 8613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:10.540871 8613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:10.542122 8613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:10.553478 8618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:10.553439 8619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:10.553514 8621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:10.554697 8613 server_base.cc:1034] running on GCE node
I20260927 22:52:10.555371 8613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:10.556648 8613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:10.557818 8613 hybrid_clock.cc:648] HybridClock initialized: now 1790549530557778 us; error 29 us; skew 500 ppm
I20260927 22:52:10.558162 8613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T22:52:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1696 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:10 dist-test-slave-df53 krb5kdc[8627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:10 dist-test-slave-df53 krb5kdc[8627](info): set up 2 sockets
Sep 27 22:52:10 dist-test-slave-df53 krb5kdc[8627](info): commencing operation
krb5kdc: starting...
W20260927 22:52:11.718509 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:11 dist-test-slave-df53 krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:11.920543 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35645
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:12.230926 8640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:12.231179 8640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:12.231262 8640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:12.238328 8640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:12.238456 8640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:12.238502 8640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:12.238556 8640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:12.238597 8640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:12.247874 8640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:12.249022 8640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:12.250238 8640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:12.261483 8645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:12.261560 8646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:12.261523 8648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:12.262250 8640 server_base.cc:1034] running on GCE node
I20260927 22:52:12.262996 8640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:12.264279 8640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:12.265499 8640 hybrid_clock.cc:648] HybridClock initialized: now 1790549532265438 us; error 56 us; skew 500 ppm
I20260927 22:52:12.265874 8640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:12 dist-test-slave-df53 krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:12.268961 8640 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:12.270469 8640 webserver.cc:460] Webserver started at http://127.0.97.62:43465/ using document root <none> and password file <none>
I20260927 22:52:12.271070 8640 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:12.271163 8640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:12.271404 8640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:12.274194 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "75b987edee1d47dd9dffb99410ce8773"
format_stamp: "Formatted at 2026-09-27 22:52:12 on dist-test-slave-df53"
server_key: "c62e9b2959060005bff3f286657920e1"
server_key_iv: "7b951f3cf7f5eb53a16a9d9d6b9d7966"
server_key_version: "encryptionkey@0"
I20260927 22:52:12.274899 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "75b987edee1d47dd9dffb99410ce8773"
format_stamp: "Formatted at 2026-09-27 22:52:12 on dist-test-slave-df53"
server_key: "c62e9b2959060005bff3f286657920e1"
server_key_iv: "7b951f3cf7f5eb53a16a9d9d6b9d7966"
server_key_version: "encryptionkey@0"
I20260927 22:52:12.279757 8640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:52:12.282974 8655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:12.284186 8640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:52:12.284360 8640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "75b987edee1d47dd9dffb99410ce8773"
format_stamp: "Formatted at 2026-09-27 22:52:12 on dist-test-slave-df53"
server_key: "c62e9b2959060005bff3f286657920e1"
server_key_iv: "7b951f3cf7f5eb53a16a9d9d6b9d7966"
server_key_version: "encryptionkey@0"
I20260927 22:52:12.284544 8640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:12.304780 8640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:12.309463 8640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:12.309791 8640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:12.327309 8640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35645
I20260927 22:52:12.327344 8706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35645 every 8 connection(s)
I20260927 22:52:12.328588 8640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:12.330377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 8640
I20260927 22:52:12.330595 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:12.331241 388 external_mini_cluster.cc:1428] Setting key ec04b103732c2a2f95d9d8ac4f530acb
I20260927 22:52:12.332413 8707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:12.346113 8707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773: Bootstrap starting.
Sep 27 22:52:12 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:12.349032 8707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:12.350067 8707 log.cc:824] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:12.352752 8707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773: No bootstrap required, opened a new log
I20260927 22:52:12.356348 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:12.333007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.0.97.62:35645)
0927 22:52:12.333756 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:12.333768 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:12.338438 (+ 4670us) server_negotiation.cc:407] Connection header received
0927 22:52:12.340228 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:12.340252 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:12.340632 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:12.341029 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:12.342461 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:12.343729 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:12.344935 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:12.345583 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:12.345822 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:12.349580 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:12.349626 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:12.349649 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:12.349689 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:12.352578 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:12.353286 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:12.353292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:12.353298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:12.353456 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:12.353844 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:12.353851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:12.353854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:12.354352 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:12.354661 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:12.355079 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 22:52:12.355440 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":366,"threads_started":1}
I20260927 22:52:12.360294 8707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } }
I20260927 22:52:12.360625 8707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:12.360723 8707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b987edee1d47dd9dffb99410ce8773, State: Initialized, Role: FOLLOWER
I20260927 22:52:12.361256 8707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } }
I20260927 22:52:12.361477 8707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:12.361630 8707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:12.361810 8707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:12.364146 8707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } }
I20260927 22:52:12.364599 8707 leader_election.cc:302] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75b987edee1d47dd9dffb99410ce8773; no voters:
I20260927 22:52:12.365113 8707 leader_election.cc:288] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:12.365233 8712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:12.367146 8712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 1 LEADER]: Becoming Leader. State: Replica: 75b987edee1d47dd9dffb99410ce8773, State: Running, Role: LEADER
I20260927 22:52:12.367416 8707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:12.367730 8712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } }
I20260927 22:52:12.370309 8714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75b987edee1d47dd9dffb99410ce8773. Latest consensus state: current_term: 1 leader_uuid: "75b987edee1d47dd9dffb99410ce8773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } } }
I20260927 22:52:12.370674 8714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:12.371165 8713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75b987edee1d47dd9dffb99410ce8773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } } }
I20260927 22:52:12.371450 8713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:12.372439 8720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:12.376926 8720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:12.384848 8720 catalog_manager.cc:1347] Generated new cluster ID: 71213e91462b43d3ab3e0f341896b226
I20260927 22:52:12.384963 8720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:12.404680 8720 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:12.406015 8720 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:12.421339 8720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773: Generated new TSK 0
I20260927 22:52:12.422374 8720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:52:12.492722 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35645
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:12.804220 8733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:12.804471 8733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:12.804563 8733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:12.811509 8733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:12.811604 8733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:12.811712 8733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:52:12.820791 8733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:12.821923 8733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:12.823071 8733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:12.835283 8738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:12.835207 8739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:12.835254 8741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:12.836041 8733 server_base.cc:1034] running on GCE node
I20260927 22:52:12.836563 8733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:12.837396 8733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:12.838611 8733 hybrid_clock.cc:648] HybridClock initialized: now 1790549532838577 us; error 47 us; skew 500 ppm
I20260927 22:52:12.838901 8733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:12 dist-test-slave-df53 krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:12.841964 8733 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:52:12.843344 8733 webserver.cc:460] Webserver started at http://127.0.97.1:35139/ using document root <none> and password file <none>
I20260927 22:52:12.843972 8733 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:12.844141 8733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:12.844439 8733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:12.847215 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "fd6dd4a552414e4fa77eaf0419f2dbbf"
format_stamp: "Formatted at 2026-09-27 22:52:12 on dist-test-slave-df53"
server_key: "b7db23547d629e2966cfea3f33d02896"
server_key_iv: "0d9137e7e7f39390934a7e3956d6c61b"
server_key_version: "encryptionkey@0"
I20260927 22:52:12.847947 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fd6dd4a552414e4fa77eaf0419f2dbbf"
format_stamp: "Formatted at 2026-09-27 22:52:12 on dist-test-slave-df53"
server_key: "b7db23547d629e2966cfea3f33d02896"
server_key_iv: "0d9137e7e7f39390934a7e3956d6c61b"
server_key_version: "encryptionkey@0"
I20260927 22:52:12.852672 8733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 22:52:12.855912 8748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:12.857244 8733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:52:12.857383 8733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "fd6dd4a552414e4fa77eaf0419f2dbbf"
format_stamp: "Formatted at 2026-09-27 22:52:12 on dist-test-slave-df53"
server_key: "b7db23547d629e2966cfea3f33d02896"
server_key_iv: "0d9137e7e7f39390934a7e3956d6c61b"
server_key_version: "encryptionkey@0"
I20260927 22:52:12.857546 8733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:12.880175 8733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:12.883947 8733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:12.884223 8733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:12.885082 8733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:12.886359 8733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:12.886431 8733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:12.886595 8733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:12.886648 8733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:12.918589 8733 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39285
I20260927 22:52:12.918676 8860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39285 every 8 connection(s)
I20260927 22:52:12.919848 8733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:52:12.923208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 8733
I20260927 22:52:12.923444 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:52:12.924223 388 external_mini_cluster.cc:1428] Setting key 9df1097e5748b4034ce5c01519fa02bc
Sep 27 22:52:12 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:52:12.944022 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:12.922425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46857 (local address 127.0.97.62:35645)
0927 22:52:12.922836 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:12.922841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:12.924044 (+ 1203us) server_negotiation.cc:407] Connection header received
0927 22:52:12.926161 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:12.926166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:12.926252 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:12.926376 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:12.927820 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:12.928525 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:12.930118 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:12.930523 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:12.930654 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:12.938179 (+ 7525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:12.938184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:12.938187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:12.938218 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:12.940839 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:12.941784 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:12.941788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:12.941790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:12.941899 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:12.942367 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:12.942370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:12.942372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:12.942624 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:12.942728 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:12.943610 (+ 882us) server_negotiation.cc:299] Negotiation successful
0927 22:52:12.943764 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":216,"threads_started":1}
I20260927 22:52:12.944953 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:12.922925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:46857)
0927 22:52:12.923854 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:12.923907 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:12.925897 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:12.926607 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:12.926616 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:12.927120 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:12.927664 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:12.927680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:12.928670 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:12.928676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:12.929974 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:12.929986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:12.930736 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:12.930745 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:12.930970 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:12.935800 (+ 4830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:12.935822 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:12.937955 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:12.940989 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:12.941002 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:12.941024 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:12.941663 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:12.942008 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:12.942013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:12.942016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:12.942254 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:12.942767 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:12.942778 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:12.943202 (+ 424us) client_negotiation.cc:769] Sending connection context
0927 22:52:12.943525 (+ 323us) client_negotiation.cc:240] Negotiation successful
0927 22:52:12.943911 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":348,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:52:12.946612 8861 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35645
I20260927 22:52:12.946913 8861 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:12.947572 8861 heartbeater.cc:511] Master 127.0.97.62:35645 requested a full tablet report, sending...
I20260927 22:52:12.949537 8672 ts_manager.cc:194] Registered new tserver with Master: fd6dd4a552414e4fa77eaf0419f2dbbf (127.0.97.1:39285)
I20260927 22:52:12.950970 8672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:52:12.969113 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35645
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:13.272686 8867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:13.272962 8867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:13.273048 8867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:13.280001 8867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:13.280092 8867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:13.280203 8867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:52:13.289554 8867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:13.290634 8867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:13.291765 8867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:13.303942 8872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:13.304019 8875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:13.303997 8873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:13.306862 8867 server_base.cc:1034] running on GCE node
I20260927 22:52:13.307408 8867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:13.308212 8867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:13.309425 8867 hybrid_clock.cc:648] HybridClock initialized: now 1790549533309388 us; error 49 us; skew 500 ppm
I20260927 22:52:13.309798 8867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:13 dist-test-slave-df53 krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:13.312650 8867 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:52:13.314106 8867 webserver.cc:460] Webserver started at http://127.0.97.2:45433/ using document root <none> and password file <none>
I20260927 22:52:13.314764 8867 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:13.314901 8867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:13.315194 8867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:13.317951 8867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "b74a82252c594da884d60d4bbe609080"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-df53"
server_key: "fb9b90d78433e69809254df40c650398"
server_key_iv: "8cf2bdd4aa3df2d8c6e8c8e375968ccb"
server_key_version: "encryptionkey@0"
I20260927 22:52:13.318707 8867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b74a82252c594da884d60d4bbe609080"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-df53"
server_key: "fb9b90d78433e69809254df40c650398"
server_key_iv: "8cf2bdd4aa3df2d8c6e8c8e375968ccb"
server_key_version: "encryptionkey@0"
I20260927 22:52:13.323386 8867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 22:52:13.326501 8882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:13.327792 8867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:52:13.327951 8867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "b74a82252c594da884d60d4bbe609080"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-df53"
server_key: "fb9b90d78433e69809254df40c650398"
server_key_iv: "8cf2bdd4aa3df2d8c6e8c8e375968ccb"
server_key_version: "encryptionkey@0"
I20260927 22:52:13.328136 8867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:13.343664 8867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:13.347334 8867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:13.347607 8867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:13.348486 8867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:13.349776 8867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:13.349874 8867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:13.350013 8867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:13.350083 8867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:13.382148 8867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44985
I20260927 22:52:13.382192 8994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44985 every 8 connection(s)
I20260927 22:52:13.383309 8867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:52:13.386579 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 8867
I20260927 22:52:13.386751 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:52:13.387255 388 external_mini_cluster.cc:1428] Setting key d1b1bafdae19ccb2230f67de264f29b2
Sep 27 22:52:13 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:13.403683 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:13.385833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54369 (local address 127.0.97.62:35645)
0927 22:52:13.385996 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:13.386000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:13.387185 (+ 1185us) server_negotiation.cc:407] Connection header received
0927 22:52:13.389158 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:13.389163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:13.389241 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:13.389353 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:13.390787 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:13.391908 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:13.393358 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:13.394135 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:13.394271 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:13.397955 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:13.397959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:13.397963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:13.397991 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:13.400493 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:13.401353 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:13.401356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:13.401358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:13.401472 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:13.402011 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:13.402016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:13.402018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:13.402269 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:13.402385 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:13.403246 (+ 861us) server_negotiation.cc:299] Negotiation successful
0927 22:52:13.403421 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:52:13.405158 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:13.386263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:54369)
0927 22:52:13.387024 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:13.387073 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:13.388856 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:13.389548 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:13.389556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:13.390092 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:13.390629 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:13.390646 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:13.392039 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:13.392043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:13.393223 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:13.393233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:13.394319 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:13.394324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:13.394502 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:13.395147 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:13.395171 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:13.397758 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:13.400631 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:13.400642 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:13.400661 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:13.401241 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:13.401586 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:13.401612 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:13.401618 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:13.401852 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:13.402403 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:13.402413 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:13.402859 (+ 446us) client_negotiation.cc:769] Sending connection context
0927 22:52:13.403148 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 22:52:13.403699 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":332,"threads_started":1}
I20260927 22:52:13.406697 8995 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35645
I20260927 22:52:13.407022 8995 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:13.407919 8995 heartbeater.cc:511] Master 127.0.97.62:35645 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:52:13.409456 8672 ts_manager.cc:194] Registered new tserver with Master: b74a82252c594da884d60d4bbe609080 (127.0.97.2:44985)
I20260927 22:52:13.410279 8672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:52:13.434628 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35645
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:13.746274 9000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:13.746562 9000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:13.746719 9000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:13.753621 9000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:13.753769 9000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:13.753926 9000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:52:13.763895 9000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:13.765121 9000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:13.766402 9000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:13.779124 9005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:13.779171 9008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:13.779227 9006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:13.780006 9000 server_base.cc:1034] running on GCE node
I20260927 22:52:13.780566 9000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:13.781468 9000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:13.782670 9000 hybrid_clock.cc:648] HybridClock initialized: now 1790549533782626 us; error 58 us; skew 500 ppm
I20260927 22:52:13.782974 9000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:13 dist-test-slave-df53 krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:13.785983 9000 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:52:13.787385 9000 webserver.cc:460] Webserver started at http://127.0.97.3:41655/ using document root <none> and password file <none>
I20260927 22:52:13.788129 9000 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:13.788272 9000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:13.788568 9000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:13.791513 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "ab4da9268ffc4e2f97f8b38a0fe1e058"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-df53"
server_key: "439a32c8761104dcc6efdbae43a41c84"
server_key_iv: "062b1618c99712e2e32b32033469db1e"
server_key_version: "encryptionkey@0"
I20260927 22:52:13.792291 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ab4da9268ffc4e2f97f8b38a0fe1e058"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-df53"
server_key: "439a32c8761104dcc6efdbae43a41c84"
server_key_iv: "062b1618c99712e2e32b32033469db1e"
server_key_version: "encryptionkey@0"
I20260927 22:52:13.797281 9000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:52:13.800673 9015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:13.802035 9000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 22:52:13.802222 9000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "ab4da9268ffc4e2f97f8b38a0fe1e058"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-df53"
server_key: "439a32c8761104dcc6efdbae43a41c84"
server_key_iv: "062b1618c99712e2e32b32033469db1e"
server_key_version: "encryptionkey@0"
I20260927 22:52:13.802430 9000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:13.817763 9000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:13.821385 9000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:13.821760 9000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:13.822628 9000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:13.823832 9000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:13.823949 9000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:13.824090 9000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:13.824180 9000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:13.857226 9000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37497
I20260927 22:52:13.857270 9127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37497 every 8 connection(s)
I20260927 22:52:13.858497 9000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:52:13.862648 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 9000
I20260927 22:52:13.862823 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:52:13.863255 388 external_mini_cluster.cc:1428] Setting key 69b018e25c3b2ef6ecc5f184698e36ae
Sep 27 22:52:13 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:13.878548 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:13.860970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53105 (local address 127.0.97.62:35645)
0927 22:52:13.861173 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:13.861177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:13.862483 (+ 1306us) server_negotiation.cc:407] Connection header received
0927 22:52:13.864723 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:13.864728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:13.864824 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:13.864944 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:13.866681 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:13.867512 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:13.869036 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:13.869451 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:13.869583 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:13.872980 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:13.872994 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:13.872998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:13.873035 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:13.875607 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:13.876286 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:13.876291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:13.876295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:13.876424 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:13.876835 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:13.876840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:13.876843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:13.877108 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:13.877258 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:13.877975 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 22:52:13.878202 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:52:13.879393 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:13.861573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:53105)
0927 22:52:13.862324 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:13.862366 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:13.864413 (+ 2047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:13.865190 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:13.865201 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:13.865989 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:13.866524 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:13.866540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:13.867655 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:13.867660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:13.868891 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:13.868902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:13.869694 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:13.869702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:13.869932 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:13.870610 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:13.870636 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:13.872793 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:13.875744 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:13.875753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:13.875769 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:13.876158 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:13.876533 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:13.876538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:13.876542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:13.876715 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:13.877231 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:13.877239 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:13.877563 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 22:52:13.877883 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 22:52:13.878386 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":277,"threads_started":1}
I20260927 22:52:13.880577 9128 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35645
I20260927 22:52:13.880833 9128 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:13.881421 9128 heartbeater.cc:511] Master 127.0.97.62:35645 requested a full tablet report, sending...
I20260927 22:52:13.882727 8672 ts_manager.cc:194] Registered new tserver with Master: ab4da9268ffc4e2f97f8b38a0fe1e058 (127.0.97.3:37497)
I20260927 22:52:13.883476 8672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53105
I20260927 22:52:13.890021 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260927 22:52:13.953845 8861 heartbeater.cc:503] Master 127.0.97.62:35645 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 22:52:14.012319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 8733
Sep 27 22:52:14 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:52:14.027621 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:14.013470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.0.97.1:39285)
0927 22:52:14.014220 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:14.014227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:14.014253 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:52:14.014317 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:14.014323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:14.014525 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:14.014671 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:14.015894 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:14.016867 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:14.018173 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:14.018704 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:14.018803 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:14.022622 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:14.022632 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:14.022646 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:14.022677 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:14.025380 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:14.026005 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:14.026012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:14.026014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:14.026151 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:14.026436 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:14.026439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:14.026441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:14.026755 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:14.026962 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:14.027178 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 22:52:14.027350 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":417,"threads_started":1}
I20260927 22:52:14.028812 8856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:14.110309 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 8733
I20260927 22:52:14.126425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 8867
Sep 27 22:52:14 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:52:14.142259 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:14.127281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.0.97.2:44985)
0927 22:52:14.127826 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:14.127832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:14.127857 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:52:14.127924 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:14.127931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:14.128137 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:14.128273 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:14.129391 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:14.130580 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:14.131770 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:14.132425 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:14.132549 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:14.136519 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:14.136529 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:14.136541 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:14.136575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:14.139442 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:14.140107 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:14.140114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:14.140117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:14.140235 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:14.140626 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:14.140631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:14.140636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:14.140961 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:14.141292 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:14.141672 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 22:52:14.141920 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":299,"threads_started":1}
I20260927 22:52:14.143446 8990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:14.223129 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 8867
I20260927 22:52:14.238231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9000
Sep 27 22:52:14 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:52:14.254212 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:14.239184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.0.97.3:37497)
0927 22:52:14.239774 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:14.239781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:14.239808 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:52:14.239866 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:14.239873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:14.240068 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:14.240255 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:14.241540 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:14.242740 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:14.243973 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:14.244508 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:14.244656 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:14.248784 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:14.248795 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:14.248807 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:14.248837 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:14.251515 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:14.252146 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:14.252154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:14.252156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:14.252274 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:14.252625 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:14.252632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:14.252635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:14.252969 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:14.253291 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:14.253561 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:52:14.253849 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":289,"threads_started":1}
I20260927 22:52:14.255407 9123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:14.336391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 9000
I20260927 22:52:14.351789 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 8640
I20260927 22:52:14.352905 8702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:14.452955 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 8640
I20260927 22:52:14.474179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35645
--webserver_interface=127.0.97.62
--webserver_port=43465
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:14.785331 9145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:14.785661 9145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:14.785825 9145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:14.792876 9145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:14.792976 9145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:14.793061 9145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:14.793174 9145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:14.793256 9145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:14.802747 9145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:14.803997 9145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:14.805292 9145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:14.816416 9153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:14.816548 9150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:14.816548 9151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:14.817375 9145 server_base.cc:1034] running on GCE node
I20260927 22:52:14.818140 9145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:14.819415 9145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:14.820676 9145 hybrid_clock.cc:648] HybridClock initialized: now 1790549534820648 us; error 39 us; skew 500 ppm
I20260927 22:52:14.821005 9145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:14 dist-test-slave-df53 krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:14.823992 9145 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:14.825253 9145 webserver.cc:460] Webserver started at http://127.0.97.62:43465/ using document root <none> and password file <none>
I20260927 22:52:14.825927 9145 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:14.826042 9145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:14.830951 9145 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:52:14.833523 9160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:14.834795 9145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:52:14.834921 9145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "75b987edee1d47dd9dffb99410ce8773"
format_stamp: "Formatted at 2026-09-27 22:52:12 on dist-test-slave-df53"
server_key: "c62e9b2959060005bff3f286657920e1"
server_key_iv: "7b951f3cf7f5eb53a16a9d9d6b9d7966"
server_key_version: "encryptionkey@0"
I20260927 22:52:14.835762 9145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:14.852421 9145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:14.856467 9145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:14.856683 9145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:14.873907 9145 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35645
I20260927 22:52:14.873939 9211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35645 every 8 connection(s)
I20260927 22:52:14.875304 9145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:14.881549 9212 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:52:14.882957 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 9145
I20260927 22:52:14.883893 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39285
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35645
--webserver_port=35139
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:52:14.885839 9212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773: Bootstrap starting.
I20260927 22:52:14.905534 9212 log.cc:824] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:14.913873 9212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:52:14.914402 9212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773: Bootstrap complete.
I20260927 22:52:14.923230 9212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } }
I20260927 22:52:14.923990 9212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b987edee1d47dd9dffb99410ce8773, State: Initialized, Role: FOLLOWER
I20260927 22:52:14.924662 9212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } }
I20260927 22:52:14.924887 9212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:14.924996 9212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:14.925184 9212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:52:14.928716 9212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } }
I20260927 22:52:14.929179 9212 leader_election.cc:302] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75b987edee1d47dd9dffb99410ce8773; no voters:
I20260927 22:52:14.929831 9212 leader_election.cc:288] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:52:14.930078 9216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:52:14.931138 9216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [term 2 LEADER]: Becoming Leader. State: Replica: 75b987edee1d47dd9dffb99410ce8773, State: Running, Role: LEADER
I20260927 22:52:14.931689 9212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:14.931847 9216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } }
I20260927 22:52:14.935086 9218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75b987edee1d47dd9dffb99410ce8773. Latest consensus state: current_term: 2 leader_uuid: "75b987edee1d47dd9dffb99410ce8773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } } }
I20260927 22:52:14.935439 9218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:14.935938 9217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "75b987edee1d47dd9dffb99410ce8773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b987edee1d47dd9dffb99410ce8773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35645 } } }
I20260927 22:52:14.936223 9217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:14.943387 9225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:14.948829 9225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:14.950630 9225 catalog_manager.cc:1259] Loaded cluster ID: 71213e91462b43d3ab3e0f341896b226
I20260927 22:52:14.950821 9225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:14.955435 9234 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:52:14.955670 9225 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:14.957255 9225 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 75b987edee1d47dd9dffb99410ce8773: Loaded TSK: 0
I20260927 22:52:14.958405 9225 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:52:15.224972 9214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:15.225301 9214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:15.225453 9214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:15.233807 9214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:15.233946 9214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:15.234105 9214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:52:15.243875 9214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35139
--tserver_master_addrs=127.0.97.62:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:15.245075 9214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:15.246577 9214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:15.260758 9239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:15.260659 9240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:15.261366 9242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:15.261576 9214 server_base.cc:1034] running on GCE node
I20260927 22:52:15.262301 9214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:15.263387 9214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:15.264621 9214 hybrid_clock.cc:648] HybridClock initialized: now 1790549535264554 us; error 58 us; skew 500 ppm
I20260927 22:52:15.264969 9214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549535, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:15.268646 9214 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 22:52:15.270159 9214 webserver.cc:460] Webserver started at http://127.0.97.1:35139/ using document root <none> and password file <none>
I20260927 22:52:15.270925 9214 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:15.271055 9214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:15.277779 9214 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 22:52:15.280903 9249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:15.282524 9214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:15.282685 9214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "fd6dd4a552414e4fa77eaf0419f2dbbf"
format_stamp: "Formatted at 2026-09-27 22:52:12 on dist-test-slave-df53"
server_key: "b7db23547d629e2966cfea3f33d02896"
server_key_iv: "0d9137e7e7f39390934a7e3956d6c61b"
server_key_version: "encryptionkey@0"
I20260927 22:52:15.283962 9214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:15.307363 9214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:15.311426 9214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:15.311699 9214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:15.312664 9214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:15.314042 9214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:15.314160 9214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:15.314299 9214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:15.314388 9214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:15.344789 9214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39285
I20260927 22:52:15.344871 9361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39285 every 8 connection(s)
I20260927 22:52:15.346243 9214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:52:15.356699 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 9214
I20260927 22:52:15.357295 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44985
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35645
--webserver_port=45433
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:52:15.365384 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:15.349209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39613 (local address 127.0.97.62:35645)
0927 22:52:15.350022 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:15.350038 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:15.350339 (+ 301us) server_negotiation.cc:407] Connection header received
0927 22:52:15.352308 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:15.352322 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:15.352633 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:15.352943 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:15.354311 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.355597 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.357189 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.358039 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.358211 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.364523 (+ 6312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39613: BlockingRecv error: recv got EOF from 127.0.97.1:39613 (error 108)
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":224,"threads_started":1}
I20260927 22:52:15.365430 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:15.349297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:39613)
0927 22:52:15.350172 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:15.350209 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:15.351825 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:15.353079 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:15.353087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:15.353575 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:15.354140 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.354156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.355747 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.355750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.357037 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.357049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.358296 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.358306 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.358631 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.359379 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:15.359417 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:15.364130 (+ 4713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":495,"threads_started":1}
W20260927 22:52:15.366080 9362 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35645 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35645: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:52:15.375077 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:15.366887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52181 (local address 127.0.97.62:35645)
0927 22:52:15.367077 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:15.367084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:15.367102 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:52:15.367152 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:15.367160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:15.367233 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:15.367337 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:15.367903 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.369121 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.370359 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.371083 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.371217 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.374929 (+ 3712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52181: BlockingRecv error: recv got EOF from 127.0.97.1:52181 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:52:15.375133 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:15.366679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:52181)
0927 22:52:15.366855 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:15.366873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:15.366970 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:15.367372 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:15.367375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:15.367640 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:15.367770 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.367777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.369261 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.369265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.370229 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.370239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.371225 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.371229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.371344 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.371897 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:15.371913 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:15.374787 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:15.384259 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:15.375806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:49153)
0927 22:52:15.375970 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:15.375990 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:15.376073 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:15.376433 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:15.376436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:15.376686 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:15.376811 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.376818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.378374 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.378378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.379271 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.379281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.380351 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.380355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.380470 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.381063 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:15.381079 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:15.383982 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:52:15.384418 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:15.375949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49153 (local address 127.0.97.62:35645)
0927 22:52:15.376119 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:15.376128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:15.376149 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:52:15.376200 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:15.376208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:15.376310 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:15.376420 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:15.376943 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.378192 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.379413 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.380193 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.380363 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.384178 (+ 3815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49153: BlockingRecv error: recv got EOF from 127.0.97.1:49153 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 22:52:15.384545 9362 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:52:15.683976 9366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:15.684219 9366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:15.684370 9366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:15.691684 9366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:15.691787 9366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:15.691984 9366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:52:15.701200 9366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45433
--tserver_master_addrs=127.0.97.62:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:15.702494 9366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:15.703697 9366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:15.719224 9372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:15.719199 9371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:15.720299 9374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:15.721128 9366 server_base.cc:1034] running on GCE node
I20260927 22:52:15.721792 9366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:15.722739 9366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:15.723958 9366 hybrid_clock.cc:648] HybridClock initialized: now 1790549535723902 us; error 57 us; skew 500 ppm
I20260927 22:52:15.724264 9366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549535, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:15.727499 9366 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 22:52:15.729038 9366 webserver.cc:460] Webserver started at http://127.0.97.2:45433/ using document root <none> and password file <none>
I20260927 22:52:15.729754 9366 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:15.729902 9366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:15.734987 9366 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 22:52:15.737668 9381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:15.739131 9366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:52:15.739283 9366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "b74a82252c594da884d60d4bbe609080"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-df53"
server_key: "fb9b90d78433e69809254df40c650398"
server_key_iv: "8cf2bdd4aa3df2d8c6e8c8e375968ccb"
server_key_version: "encryptionkey@0"
I20260927 22:52:15.740260 9366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:15.753273 9366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:15.758973 9366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:15.759356 9366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:15.760675 9366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:15.762072 9366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:15.762195 9366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:15.762318 9366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:15.762413 9366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:15.800138 9366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44985
I20260927 22:52:15.800223 9493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44985 every 8 connection(s)
I20260927 22:52:15.801563 9366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:52:15.810954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 9366
I20260927 22:52:15.811549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37497
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35645
--webserver_port=41655
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:52:15.817246 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:15.804111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49531 (local address 127.0.97.62:35645)
0927 22:52:15.804282 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:15.804287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:15.805382 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 22:52:15.807185 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:15.807190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:15.807252 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:15.807339 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:15.808745 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.809437 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.810987 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.811452 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.811584 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.817069 (+ 5485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49531: BlockingRecv error: recv got EOF from 127.0.97.2:49531 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:52:15.818179 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:15.804550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:49531)
0927 22:52:15.805215 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:15.805260 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:15.806911 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:15.807549 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:15.807557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:15.808048 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:15.808588 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.808603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.809558 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.809562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.810837 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.810847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.811664 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.811671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.811940 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.812673 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:15.812703 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:15.816929 (+ 4226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":269,"threads_started":1}
W20260927 22:52:15.818925 9494 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35645 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35645: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:52:15.828672 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:15.819573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36501 (local address 127.0.97.62:35645)
0927 22:52:15.819730 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:15.819734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:15.819749 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:52:15.819900 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:15.819903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:15.819955 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:15.820030 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:15.820842 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.821512 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.823466 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.823863 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.823966 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.828502 (+ 4536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36501: BlockingRecv error: recv got EOF from 127.0.97.2:36501 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:52:15.828676 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:15.819487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:36501)
0927 22:52:15.819668 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:15.819689 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:15.819785 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:15.820078 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:15.820085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:15.820518 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:15.820718 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.820731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.821667 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.821675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.823336 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.823352 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.824020 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.824029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.824255 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.824807 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:15.824833 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:15.828317 (+ 3484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:15 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:15.836782 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:15.829433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:52147)
0927 22:52:15.829635 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:15.829654 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:15.829772 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:15.830082 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:15.830086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:15.830374 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:15.830506 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.830513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.831458 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.831463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.832341 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:15.832351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.833115 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.833120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:15.833301 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.833926 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:15.833942 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:15.836478 (+ 2536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:52:15.836865 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:15.829527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52147 (local address 127.0.97.62:35645)
0927 22:52:15.829718 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:15.829724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:15.829741 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:52:15.829904 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:15.829915 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:15.829968 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:15.830064 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:15.830640 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.831340 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.832563 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:15.832966 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:15.833094 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:15.836660 (+ 3566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52147: BlockingRecv error: recv got EOF from 127.0.97.2:52147 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 22:52:15.837183 9494 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:52:16.127084 9497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:16.127382 9497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:16.127550 9497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:16.134750 9497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:16.134902 9497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:16.135084 9497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:52:16.144749 9497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41655
--tserver_master_addrs=127.0.97.62:35645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:16.146188 9497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:16.147454 9497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:16.160043 9503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:16.160092 9505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:16.160130 9502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:16.161056 9497 server_base.cc:1034] running on GCE node
I20260927 22:52:16.161717 9497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:16.162644 9497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:16.163836 9497 hybrid_clock.cc:648] HybridClock initialized: now 1790549536163780 us; error 52 us; skew 500 ppm
I20260927 22:52:16.164129 9497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549536, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:16.167235 9497 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 22:52:16.168646 9497 webserver.cc:460] Webserver started at http://127.0.97.3:41655/ using document root <none> and password file <none>
I20260927 22:52:16.169338 9497 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:16.169446 9497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:16.174680 9497 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:52:16.177207 9512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:16.178588 9497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:52:16.178715 9497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "ab4da9268ffc4e2f97f8b38a0fe1e058"
format_stamp: "Formatted at 2026-09-27 22:52:13 on dist-test-slave-df53"
server_key: "439a32c8761104dcc6efdbae43a41c84"
server_key_iv: "062b1618c99712e2e32b32033469db1e"
server_key_version: "encryptionkey@0"
I20260927 22:52:16.179560 9497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:16.201934 9497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:16.205718 9497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:16.205938 9497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:16.206801 9497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:16.208000 9497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:16.208084 9497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:16.208151 9497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:16.208205 9497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:16.240504 9497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37497
I20260927 22:52:16.240592 9624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37497 every 8 connection(s)
I20260927 22:52:16.242041 9497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:52:16.245427 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 9497
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:52:16.258503 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:16.244556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44831 (local address 127.0.97.62:35645)
0927 22:52:16.244765 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:16.244770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:16.246180 (+ 1410us) server_negotiation.cc:407] Connection header received
0927 22:52:16.248288 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:16.248300 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:16.248409 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:16.248521 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:16.250160 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.250955 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.252558 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.253124 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.253238 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.258214 (+ 4976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44831: BlockingRecv error: recv got EOF from 127.0.97.3:44831 (error 108)
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":55}
I20260927 22:52:16.259200 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:16.245149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:44831)
0927 22:52:16.245996 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:16.246053 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:16.248010 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:16.248785 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:16.248798 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:16.249385 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:16.249993 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:16.250010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.251092 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.251097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:16.252393 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:16.252432 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.253344 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.253351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:16.253532 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.254211 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:16.254231 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:16.258035 (+ 3804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":508,"threads_started":1}
W20260927 22:52:16.259975 9625 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35645 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:35645: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260927 22:52:16.262871 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:16.246481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.0.97.62:35645)
0927 22:52:16.248603 (+ 2122us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:16.248608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:16.248623 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:52:16.248669 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:16.248673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:16.248736 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:16.248841 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:16.250398 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.251251 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.252476 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.253036 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.253136 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.255825 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:16.255833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:16.255855 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:16.255886 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:16.258992 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:16.259790 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:16.259799 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:16.259807 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:16.260023 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:16.260402 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:16.260406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:16.260408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:16.260987 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:16.261260 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:16.262170 (+ 910us) server_negotiation.cc:299] Negotiation successful
0927 22:52:16.262473 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":2013,"thread_start_us":1879,"threads_started":1}
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:16.268299 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:16.260585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:58069)
0927 22:52:16.260733 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:16.260752 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:16.260828 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:16.261109 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:16.261111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:16.261371 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:16.261496 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:16.261502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.262725 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.262729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:16.263714 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:16.263724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.264388 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.264393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:16.264553 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.265408 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:16.265434 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:16.268087 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:52:16.268428 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:16.260667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58069 (local address 127.0.97.62:35645)
0927 22:52:16.260812 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:16.260816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:16.260832 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:52:16.260937 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:16.260941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:16.260999 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:16.261085 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:16.261646 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.262585 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.263870 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.264255 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.264339 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.268242 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58069: BlockingRecv error: recv got EOF from 127.0.97.3:58069 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:16.275858 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:16.268977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:42411)
0927 22:52:16.269110 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:16.269123 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:16.269227 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:16.269574 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:16.269578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:16.269883 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:16.270008 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:16.270015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.270956 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.270963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:16.271834 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:16.271843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.272501 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.272506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:16.272666 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.273108 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:16.273123 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:16.275581 (+ 2458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:52:16.275972 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:16.269066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42411 (local address 127.0.97.62:35645)
0927 22:52:16.269300 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:16.269308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:16.269324 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:52:16.269396 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:16.269401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:16.269459 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:16.269553 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:16.270135 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.270827 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.271972 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.272359 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.272494 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.275736 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42411: BlockingRecv error: recv got EOF from 127.0.97.3:42411 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260927 22:52:16.276211 9625 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:16.393225 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:16.385060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:52837)
0927 22:52:16.385565 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:16.385581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:16.385704 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:16.386034 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:16.386037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:16.386455 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:16.386643 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:16.386651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.387685 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.387690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:16.388801 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:16.388813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.389500 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.389505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:16.389711 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.390373 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:16.390390 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:16.393015 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":273,"threads_started":1}
W20260927 22:52:16.393548 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:16.385148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52837 (local address 127.0.97.62:35645)
0927 22:52:16.385319 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:16.385323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:16.385676 (+ 353us) server_negotiation.cc:407] Connection header received
0927 22:52:16.385839 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:16.385843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:16.385900 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:16.385977 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:16.386801 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.387537 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.388933 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.389356 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.389495 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.393209 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52837: BlockingRecv error: recv got EOF from 127.0.97.1:52837 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:16 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:16.846205 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:16.837753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:51731)
0927 22:52:16.838199 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:16.838215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:16.838334 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:16.838677 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:16.838680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:16.839142 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:16.839320 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:16.839329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.840355 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.840359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:16.841294 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:16.841305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.842023 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.842028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:16.842148 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.842829 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:16.842847 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:16.845917 (+ 3070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
W20260927 22:52:16.846444 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:16.837784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51731 (local address 127.0.97.62:35645)
0927 22:52:16.838028 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:16.838035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:16.838308 (+ 273us) server_negotiation.cc:407] Connection header received
0927 22:52:16.838440 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:16.838443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:16.838502 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:16.838580 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:16.839443 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.840234 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.841435 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:16.841879 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:16.841971 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:16.846150 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51731: BlockingRecv error: recv got EOF from 127.0.97.2:51731 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 22:52:17 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:17 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:17.285730 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:17.276823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:37263)
0927 22:52:17.277325 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:17.277345 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:17.277502 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:17.278015 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:17.278019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:17.278452 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:17.278634 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:17.278642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.279632 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.279637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:17.280601 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:17.280610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.281349 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.281355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:17.281527 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:17.282100 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:17.282117 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:17.285355 (+ 3238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":272,"threads_started":1}
W20260927 22:52:17.285848 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:17.276911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37263 (local address 127.0.97.62:35645)
0927 22:52:17.277149 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:17.277158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:17.277428 (+ 270us) server_negotiation.cc:407] Connection header received
0927 22:52:17.277650 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:17.277655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:17.277727 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:17.277773 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:17.278777 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.279485 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.280751 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.281169 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.281296 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:17.285577 (+ 4281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37263: BlockingRecv error: recv got EOF from 127.0.97.3:37263 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 22:52:17 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:17 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:17.402015 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:17.394203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:40563)
0927 22:52:17.394664 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:17.394681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:17.394793 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:17.395165 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:17.395168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:17.395536 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:17.395676 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:17.395683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.396663 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.396666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:17.397568 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:17.397579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.398259 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.398263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:17.398366 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:17.399025 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:17.399042 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:17.401738 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":263,"threads_started":1}
W20260927 22:52:17.402302 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:17.394259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40563 (local address 127.0.97.62:35645)
0927 22:52:17.394513 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:17.394521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:17.394762 (+ 241us) server_negotiation.cc:407] Connection header received
0927 22:52:17.394938 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:17.394942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:17.395029 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:17.395115 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:17.395803 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.396545 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.397703 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.398120 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.398220 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:17.402012 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40563: BlockingRecv error: recv got EOF from 127.0.97.1:40563 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:52:17 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:17 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:17.854971 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:17.847013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:45667)
0927 22:52:17.847479 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:17.847494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:17.847615 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:17.847906 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:17.847909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:17.848287 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:17.848426 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:17.848434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.849541 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.849545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:17.850448 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:17.850458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.851089 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.851093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:17.851201 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:17.851837 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:17.851854 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:17.854588 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":267,"threads_started":1}
W20260927 22:52:17.854991 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:17.847103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45667 (local address 127.0.97.62:35645)
0927 22:52:17.847328 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:17.847335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:17.847572 (+ 237us) server_negotiation.cc:407] Connection header received
0927 22:52:17.847715 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:17.847719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:17.847790 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:17.847886 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:17.848576 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.849424 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.850561 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:17.850968 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:17.851058 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:17.854770 (+ 3712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45667: BlockingRecv error: recv got EOF from 127.0.97.2:45667 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 22:52:18 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:18 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:18.297550 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:18.286542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:34971)
0927 22:52:18.287016 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:18.287059 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:18.287167 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:18.287445 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:18.287448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:18.287836 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:18.287977 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:18.287985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.289010 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.289014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:18.290107 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:18.290119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.290725 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.290729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:18.290828 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:18.294637 (+ 3809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:18.294655 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:18.297323 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":285,"threads_started":1}
W20260927 22:52:18.297858 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:18.286723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34971 (local address 127.0.97.62:35645)
0927 22:52:18.286940 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:18.286946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:18.287134 (+ 188us) server_negotiation.cc:407] Connection header received
0927 22:52:18.287260 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:18.287264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:18.287322 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:18.287402 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:18.288103 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.288897 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.290229 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.290604 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.290696 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:18.297519 (+ 6823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34971: BlockingRecv error: recv got EOF from 127.0.97.3:34971 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 22:52:18 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:18 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:18.410768 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:18.402989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:43225)
0927 22:52:18.403499 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:18.403515 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:18.403627 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:18.403899 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:18.403902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:18.404214 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:18.404374 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:18.404382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.405330 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.405333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:18.406321 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:18.406333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.406938 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.406941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:18.407074 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:18.407720 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:18.407737 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:18.410529 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
W20260927 22:52:18.411051 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:18.403086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43225 (local address 127.0.97.62:35645)
0927 22:52:18.403302 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:18.403307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:18.403595 (+ 288us) server_negotiation.cc:407] Connection header received
0927 22:52:18.403720 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:18.403724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:18.403783 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:18.403867 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:18.404495 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.405202 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.406444 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.406821 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.406911 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:18.410734 (+ 3823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43225: BlockingRecv error: recv got EOF from 127.0.97.1:43225 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 22:52:18 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:18 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:18.864003 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:18.855841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:38711)
0927 22:52:18.856357 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:18.856373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:18.856551 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:18.856893 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:18.856898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:18.857286 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:18.857454 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:18.857463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.858503 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.858511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:18.859375 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:18.859385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.860187 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.860191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:18.860318 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:18.860996 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:18.861022 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:18.863766 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":277,"threads_started":1}
W20260927 22:52:18.864288 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:18.855929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38711 (local address 127.0.97.62:35645)
0927 22:52:18.856175 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:18.856184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:18.856471 (+ 287us) server_negotiation.cc:407] Connection header received
0927 22:52:18.856677 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:18.856682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:18.856757 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:18.856859 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:18.857624 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.858365 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.859526 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:18.860037 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:18.860164 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:18.864036 (+ 3872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38711: BlockingRecv error: recv got EOF from 127.0.97.2:38711 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 22:52:19 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:19 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:19.307268 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:19.298535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:57791)
0927 22:52:19.299048 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:19.299068 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:19.299184 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:19.299506 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:19.299510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:19.299828 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:19.300001 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:19.300009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.300953 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.300957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:19.302282 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:19.302303 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.303178 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.303183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:19.303365 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:19.303908 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:19.303924 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:19.306931 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":270,"threads_started":1}
W20260927 22:52:19.307389 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:19.298606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57791 (local address 127.0.97.62:35645)
0927 22:52:19.298775 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:19.298779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:19.299152 (+ 373us) server_negotiation.cc:407] Connection header received
0927 22:52:19.299281 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:19.299284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:19.299342 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:19.299422 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:19.300133 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.300821 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.302435 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.303036 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.303169 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:19.307147 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57791: BlockingRecv error: recv got EOF from 127.0.97.3:57791 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 22:52:19 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:19 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:19.419107 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:19.411662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:55855)
0927 22:52:19.412101 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:19.412124 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:19.412231 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:19.412518 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:19.412521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:19.412845 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:19.412984 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:19.412992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.413938 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.413942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:19.414880 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:19.414890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.415536 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.415540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:19.415666 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:19.416287 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:19.416305 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:19.418892 (+ 2587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":243,"threads_started":1}
W20260927 22:52:19.419591 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:19.411733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55855 (local address 127.0.97.62:35645)
0927 22:52:19.411981 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:19.411988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:19.412202 (+ 214us) server_negotiation.cc:407] Connection header received
0927 22:52:19.412337 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:19.412341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:19.412399 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:19.412478 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:19.413128 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.413826 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.415003 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.415421 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.415509 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:19.419099 (+ 3590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55855: BlockingRecv error: recv got EOF from 127.0.97.1:55855 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 22:52:19 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:19 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:19.873138 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:19.865088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:36533)
0927 22:52:19.865705 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:19.865727 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:19.865840 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:19.866166 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:19.866169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:19.866504 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:19.866657 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:19.866665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.867626 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.867630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:19.868560 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:19.868570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.869324 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.869328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:19.869448 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:19.870122 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:19.870139 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:19.872824 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":303,"threads_started":1}
W20260927 22:52:19.873281 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:19.865279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36533 (local address 127.0.97.62:35645)
0927 22:52:19.865480 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:19.865486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:19.865805 (+ 319us) server_negotiation.cc:407] Connection header received
0927 22:52:19.865937 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:19.865940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:19.865999 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:19.866123 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:19.866772 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.867481 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.868724 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:19.869196 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:19.869301 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:19.873001 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36533: BlockingRecv error: recv got EOF from 127.0.97.2:36533 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 22:52:20 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:20 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:20.316419 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:20.308247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:46793)
0927 22:52:20.308742 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:20.308761 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:20.308915 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:20.309260 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:20.309265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:20.309660 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:20.309826 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:20.309835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.310888 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.310895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:20.311847 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:20.311857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.312537 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.312543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:20.312735 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:20.313312 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:20.313328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:20.316096 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":258,"threads_started":1}
W20260927 22:52:20.316578 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:20.308310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46793 (local address 127.0.97.62:35645)
0927 22:52:20.308528 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:20.308536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:20.308847 (+ 311us) server_negotiation.cc:407] Connection header received
0927 22:52:20.309037 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:20.309042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:20.309117 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:20.309237 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:20.309970 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.310753 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.311985 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.312398 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.312528 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:20.316304 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46793: BlockingRecv error: recv got EOF from 127.0.97.3:46793 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 22:52:20 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:20 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:20.427950 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:20.420030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:58169)
0927 22:52:20.420556 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:20.420572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:20.420681 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:20.420948 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:20.420951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:20.421476 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:20.421706 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:20.421716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.422708 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.422715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:20.423603 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:20.423613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.424302 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.424308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:20.424466 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:20.425010 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:20.425026 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:20.427607 (+ 2581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":282,"threads_started":1}
W20260927 22:52:20.428020 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:20.420124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58169 (local address 127.0.97.62:35645)
0927 22:52:20.420298 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:20.420302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:20.420643 (+ 341us) server_negotiation.cc:407] Connection header received
0927 22:52:20.420769 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:20.420773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:20.420828 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:20.420906 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:20.421850 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.422565 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.423740 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.424159 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.424295 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:20.427801 (+ 3506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58169: BlockingRecv error: recv got EOF from 127.0.97.1:58169 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 22:52:20 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:20 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:20.882580 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:20.874082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:34641)
0927 22:52:20.874715 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:20.874730 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:20.874885 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:20.875210 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:20.875215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:20.875624 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:20.875780 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:20.875789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.876805 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.876811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:20.877826 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:20.877837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.878594 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.878600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:20.878780 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:20.879394 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:20.879412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:20.882158 (+ 2746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":372,"threads_started":1}
W20260927 22:52:20.882638 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:20.874266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34641 (local address 127.0.97.62:35645)
0927 22:52:20.874485 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:20.874494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:20.874821 (+ 327us) server_negotiation.cc:407] Connection header received
0927 22:52:20.875009 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:20.875014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:20.875082 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:20.875182 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:20.875923 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.876656 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.877965 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:20.878454 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:20.878585 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:20.882365 (+ 3780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34641: BlockingRecv error: recv got EOF from 127.0.97.2:34641 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:52:21 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:21 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:21.325302 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:21.317413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:35387)
0927 22:52:21.317929 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:21.317947 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:21.318088 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:21.318426 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:21.318431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:21.318831 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:21.318978 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:21.318986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.319973 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.319979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:21.320939 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:21.320949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.321648 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.321654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:21.321824 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:21.322394 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:21.322410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:21.325077 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":254,"threads_started":1}
W20260927 22:52:21.325532 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:21.317488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35387 (local address 127.0.97.62:35645)
0927 22:52:21.317739 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:21.317746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:21.318032 (+ 286us) server_negotiation.cc:407] Connection header received
0927 22:52:21.318219 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:21.318224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:21.318293 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:21.318398 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:21.319120 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.319832 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.321076 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.321476 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.321628 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:21.325303 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35387: BlockingRecv error: recv got EOF from 127.0.97.3:35387 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 22:52:21 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:21 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:21.437041 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:21.428930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:54233)
0927 22:52:21.429379 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:21.429395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:21.429501 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:21.429824 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:21.429827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:21.430166 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:21.430312 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:21.430319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.431239 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.431242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:21.432166 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:21.432177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.432998 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.433002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:21.433124 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:21.433787 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:21.433804 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:21.436712 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":247,"threads_started":1}
W20260927 22:52:21.437145 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:21.428952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54233 (local address 127.0.97.62:35645)
0927 22:52:21.429170 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:21.429180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:21.429472 (+ 292us) server_negotiation.cc:407] Connection header received
0927 22:52:21.429609 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:21.429613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:21.429703 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:21.429778 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:21.430421 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.431123 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.432295 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.432867 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.432966 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:21.436892 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54233: BlockingRecv error: recv got EOF from 127.0.97.1:54233 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 22:52:21 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:21 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:21.892086 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:21.883550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:46751)
0927 22:52:21.884053 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:21.884074 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:21.884213 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:21.884573 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:21.884577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:21.884977 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:21.885116 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:21.885124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.886202 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.886208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:21.887112 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:21.887121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.887856 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.887862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:21.888035 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:21.888743 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:21.888769 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:21.891765 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":270,"threads_started":1}
W20260927 22:52:21.892225 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:21.883596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46751 (local address 127.0.97.62:35645)
0927 22:52:21.883838 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:21.883847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:21.884141 (+ 294us) server_negotiation.cc:407] Connection header received
0927 22:52:21.884339 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:21.884344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:21.884418 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:21.884540 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:21.885272 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.886054 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.887256 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:21.887695 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:21.887843 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:21.891984 (+ 4141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46751: BlockingRecv error: recv got EOF from 127.0.97.2:46751 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 22:52:22 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:22 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:22.334440 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:22.326290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:39311)
0927 22:52:22.326772 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:22.326792 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:22.326946 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:22.327265 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:22.327269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:22.327649 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:22.327793 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:22.327802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.328798 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.328805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:22.329761 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:22.329772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.330491 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.330497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:22.330673 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:22.331222 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:22.331238 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:22.334096 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
W20260927 22:52:22.334534 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:22.326312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39311 (local address 127.0.97.62:35645)
0927 22:52:22.326496 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:22.326505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:22.326875 (+ 370us) server_negotiation.cc:407] Connection header received
0927 22:52:22.327077 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:22.327082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:22.327137 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:22.327236 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:22.327932 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.328657 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.329897 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.330326 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.330484 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:22.334302 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39311: BlockingRecv error: recv got EOF from 127.0.97.3:39311 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 22:52:22 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:22 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:22.446590 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:22.438043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:37583)
0927 22:52:22.438680 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:22.438703 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:22.438877 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:22.439169 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:22.439172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:22.439619 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:22.439812 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:22.439820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.440760 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.440764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:22.441787 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:22.441798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.442605 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.442609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:22.442723 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:22.443385 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:22.443402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:22.446245 (+ 2843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":301,"threads_started":1}
W20260927 22:52:22.446674 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:22.438102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37583 (local address 127.0.97.62:35645)
0927 22:52:22.438306 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:22.438316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:22.438834 (+ 518us) server_negotiation.cc:407] Connection header received
0927 22:52:22.438998 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:22.439001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:22.439074 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:22.439150 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:22.439930 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.440626 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.441914 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.442475 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.442574 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:22.446442 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37583: BlockingRecv error: recv got EOF from 127.0.97.1:37583 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 22:52:22 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:22 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:22.901109 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:22.893043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:41325)
0927 22:52:22.893724 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:22.893739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:22.893880 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:22.894235 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:22.894238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:22.894631 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:22.894788 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:22.894797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.895875 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.895879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:22.896772 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:22.896782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.897494 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.897498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:22.897629 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:22.898245 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:22.898262 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:22.900868 (+ 2606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":271,"threads_started":1}
W20260927 22:52:22.901309 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:22.893096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41325 (local address 127.0.97.62:35645)
0927 22:52:22.893305 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:22.893312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:22.893823 (+ 511us) server_negotiation.cc:407] Connection header received
0927 22:52:22.894030 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:22.894038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:22.894114 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:22.894190 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:22.894939 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.895743 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.896900 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:22.897371 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:22.897466 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:22.901047 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41325: BlockingRecv error: recv got EOF from 127.0.97.2:41325 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 22:52:23 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:23 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:23.350418 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:23.335324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:40589)
0927 22:52:23.335943 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:23.335963 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:23.336075 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:23.336440 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:23.336444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:23.336979 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:23.337220 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:23.337237 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.338588 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.338592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:23.339564 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:23.339576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.340441 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.340446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:23.340550 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:23.346888 (+ 6338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:23.346906 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:23.350177 (+ 3271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":280,"threads_started":1}
W20260927 22:52:23.350557 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:23.335417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40589 (local address 127.0.97.62:35645)
0927 22:52:23.335603 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:23.335607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:23.336055 (+ 448us) server_negotiation.cc:407] Connection header received
0927 22:52:23.336200 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:23.336205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:23.336283 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:23.336392 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:23.337372 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.338448 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.339713 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.340287 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.340392 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:23.350357 (+ 9965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40589: BlockingRecv error: recv got EOF from 127.0.97.3:40589 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:52:23 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:23 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:23.455103 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:23.447363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:53055)
0927 22:52:23.447829 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:23.447845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:23.447955 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:23.448236 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:23.448239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:23.448622 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:23.448768 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:23.448776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.449778 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.449783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:23.450677 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:23.450687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.451408 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.451413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:23.451524 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:23.452094 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:23.452111 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:23.454759 (+ 2648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":265,"threads_started":1}
W20260927 22:52:23.455263 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:23.447540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53055 (local address 127.0.97.62:35645)
0927 22:52:23.447707 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:23.447712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:23.447916 (+ 204us) server_negotiation.cc:407] Connection header received
0927 22:52:23.448047 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:23.448051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:23.448121 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:23.448200 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:23.448896 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.449650 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.450811 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.451229 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.451324 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:23.454993 (+ 3669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53055: BlockingRecv error: recv got EOF from 127.0.97.1:53055 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 22:52:23 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:23 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:23.910272 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:23.902017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:58989)
0927 22:52:23.902485 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:23.902510 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:23.902618 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:23.902911 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:23.902915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:23.903301 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:23.903446 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:23.903454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.904558 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.904562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:23.905487 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:23.905498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.906357 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.906362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:23.906487 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:23.907046 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:23.907062 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:23.909953 (+ 2891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":259,"threads_started":1}
W20260927 22:52:23.910355 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:23.902109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58989 (local address 127.0.97.62:35645)
0927 22:52:23.902292 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:23.902297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:23.902585 (+ 288us) server_negotiation.cc:407] Connection header received
0927 22:52:23.902711 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:23.902714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:23.902771 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:23.902862 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:23.903575 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.904426 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.905635 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:23.906150 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:23.906248 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:23.910155 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58989: BlockingRecv error: recv got EOF from 127.0.97.2:58989 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 22:52:24 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:24 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:24.359352 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:24.351275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:60821)
0927 22:52:24.351726 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:24.351746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:24.351894 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:24.352230 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:24.352236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:24.352659 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:24.352809 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:24.352817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.353901 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.353907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:24.354807 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:24.354817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.355555 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.355561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:24.355730 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:24.356257 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:24.356273 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:24.359149 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":249,"threads_started":1}
W20260927 22:52:24.359650 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:24.351342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60821 (local address 127.0.97.62:35645)
0927 22:52:24.351575 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:24.351583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:24.351827 (+ 244us) server_negotiation.cc:407] Connection header received
0927 22:52:24.352037 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:24.352042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:24.352107 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:24.352205 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:24.352948 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.353755 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.354947 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.355370 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.355525 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:24.359336 (+ 3811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60821: BlockingRecv error: recv got EOF from 127.0.97.3:60821 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 22:52:24 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:24 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:24.464560 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:24.456009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:58993)
0927 22:52:24.456492 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:24.456508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:24.456662 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:24.457007 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:24.457011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:24.457392 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:24.457560 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:24.457568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.458647 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.458653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:24.459577 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:24.459588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.460295 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.460301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:24.460476 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:24.461151 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:24.461169 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:24.464219 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":247,"threads_started":1}
W20260927 22:52:24.464627 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:24.456063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58993 (local address 127.0.97.62:35645)
0927 22:52:24.456286 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:24.456294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:24.456593 (+ 299us) server_negotiation.cc:407] Connection header received
0927 22:52:24.456783 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:24.456788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:24.456874 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:24.456980 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:24.457754 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.458506 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.459715 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.460146 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.460305 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:24.464397 (+ 4092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58993: BlockingRecv error: recv got EOF from 127.0.97.1:58993 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 22:52:24 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:24 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:24.919407 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:24.911163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:37909)
0927 22:52:24.911647 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:24.911662 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:24.911819 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:24.912172 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:24.912176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:24.912583 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:24.912738 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:24.912746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.913827 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.913834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:24.914780 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:24.914790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.915529 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.915535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:24.915720 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:24.916336 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:24.916353 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:24.919068 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":258,"threads_started":1}
W20260927 22:52:24.919503 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:24.911222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37909 (local address 127.0.97.62:35645)
0927 22:52:24.911455 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:24.911464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:24.911749 (+ 285us) server_negotiation.cc:407] Connection header received
0927 22:52:24.911970 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:24.911974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:24.912032 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:24.912136 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:24.912905 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.913705 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.914917 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:24.915367 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:24.915499 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:24.919257 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37909: BlockingRecv error: recv got EOF from 127.0.97.2:37909 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 22:52:25 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:25 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:25.368019 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:25.360175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:38417)
0927 22:52:25.360691 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:25.360710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:25.360819 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:25.361107 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:25.361110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:25.361462 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:25.361670 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:25.361679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.362617 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.362621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:25.363522 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:25.363531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.364189 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.364193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:25.364307 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:25.364904 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:25.364955 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:25.367791 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":285,"threads_started":1}
W20260927 22:52:25.368218 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:25.360238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38417 (local address 127.0.97.62:35645)
0927 22:52:25.360409 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:25.360414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:25.360824 (+ 410us) server_negotiation.cc:407] Connection header received
0927 22:52:25.360914 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:25.360917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:25.360977 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:25.361067 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:25.361786 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.362455 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.363647 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.364067 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.364157 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:25.367979 (+ 3822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38417: BlockingRecv error: recv got EOF from 127.0.97.3:38417 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 22:52:25 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:25 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:25.472999 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:25.465454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:34665)
0927 22:52:25.465979 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:25.465994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:25.466115 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:25.466406 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:25.466409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:25.466772 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:25.466909 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:25.466916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.467926 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.467929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:25.468826 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:25.468836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.469497 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.469501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:25.469644 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:25.470167 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:25.470183 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:25.472796 (+ 2613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":319,"threads_started":1}
W20260927 22:52:25.473172 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:25.465558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34665 (local address 127.0.97.62:35645)
0927 22:52:25.465743 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:25.465747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:25.466112 (+ 365us) server_negotiation.cc:407] Connection header received
0927 22:52:25.466198 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:25.466201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:25.466281 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:25.466362 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:25.467039 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.467821 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.468962 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.469378 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.469467 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:25.472946 (+ 3479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34665: BlockingRecv error: recv got EOF from 127.0.97.1:34665 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 22:52:25 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:25 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:25.927944 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:25.920268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:46125)
0927 22:52:25.920730 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:25.920746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:25.920854 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:25.921161 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:25.921165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:25.921557 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:25.921739 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:25.921747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.922736 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.922740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:25.923658 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:25.923668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.924332 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.924337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:25.924452 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:25.925100 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:25.925117 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:25.927736 (+ 2619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":270,"threads_started":1}
W20260927 22:52:25.928154 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:25.920434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46125 (local address 127.0.97.62:35645)
0927 22:52:25.920645 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:25.920651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:25.920821 (+ 170us) server_negotiation.cc:407] Connection header received
0927 22:52:25.920948 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:25.920951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:25.921007 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:25.921087 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:25.921872 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.922587 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.923777 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:25.924201 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:25.924292 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:25.927899 (+ 3607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46125: BlockingRecv error: recv got EOF from 127.0.97.2:46125 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 22:52:26 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:26 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:26.377331 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:26.369005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:53097)
0927 22:52:26.369561 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:26.369582 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:26.369752 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:26.370081 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:26.370086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:26.370491 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:26.370634 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:26.370642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.371651 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.371658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:26.372567 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:26.372577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.373290 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.373297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:26.373469 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:26.374074 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:26.374091 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:26.377126 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":283,"threads_started":1}
W20260927 22:52:26.377569 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:26.369141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53097 (local address 127.0.97.62:35645)
0927 22:52:26.369418 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:26.369424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:26.369690 (+ 266us) server_negotiation.cc:407] Connection header received
0927 22:52:26.369876 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:26.369881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:26.369945 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:26.370057 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:26.370787 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.371508 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.372710 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.373126 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.373258 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:26.377292 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53097: BlockingRecv error: recv got EOF from 127.0.97.3:53097 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 27 22:52:26 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:26 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:26.482273 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:26.474379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:40071)
0927 22:52:26.474878 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:26.474895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:26.475006 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:26.475382 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:26.475385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:26.475755 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:26.475915 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:26.475923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.476904 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.476908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:26.477875 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:26.477887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.478527 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.478531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:26.478640 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:26.479207 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:26.479225 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:26.482040 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":289,"threads_started":1}
W20260927 22:52:26.482486 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:26.474449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40071 (local address 127.0.97.62:35645)
0927 22:52:26.474650 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:26.474656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:26.474977 (+ 321us) server_negotiation.cc:407] Connection header received
0927 22:52:26.475144 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:26.475148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:26.475252 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:26.475339 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:26.476045 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.476792 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.477988 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.478403 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.478492 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:26.482237 (+ 3745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40071: BlockingRecv error: recv got EOF from 127.0.97.1:40071 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 22:52:26 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:26 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:26.937628 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:26.928892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:52995)
0927 22:52:26.929455 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:26.929473 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:26.929620 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:26.929919 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:26.929922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:26.930349 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:26.930529 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:26.930538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.931956 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.931961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:26.933305 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:26.933317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.933997 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.934001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:26.934110 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:26.934644 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:26.934660 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:26.937370 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":312,"threads_started":1}
W20260927 22:52:26.937886 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:26.928953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52995 (local address 127.0.97.62:35645)
0927 22:52:26.929145 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:26.929149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:26.929557 (+ 408us) server_negotiation.cc:407] Connection header received
0927 22:52:26.929726 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:26.929729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:26.929789 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:26.929868 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:26.930824 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.931745 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.933420 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:26.933873 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:26.933967 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:26.937546 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52995: BlockingRecv error: recv got EOF from 127.0.97.2:52995 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 22:52:27 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:27 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:27.386137 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:27.378239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:33209)
0927 22:52:27.378790 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:27.378812 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:27.378922 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:27.379247 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:27.379251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:27.379635 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:27.379793 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:27.379801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.380735 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.380739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:27.381615 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:27.381656 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.382300 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.382304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:27.382438 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:27.383112 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:27.383140 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:27.385920 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":298,"threads_started":1}
W20260927 22:52:27.386323 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:27.378358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33209 (local address 127.0.97.62:35645)
0927 22:52:27.378527 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:27.378531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:27.378909 (+ 378us) server_negotiation.cc:407] Connection header received
0927 22:52:27.379023 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:27.379027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:27.379085 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:27.379168 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:27.379918 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.380593 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.381773 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.382145 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.382233 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:27.386115 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33209: BlockingRecv error: recv got EOF from 127.0.97.3:33209 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 22:52:27 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:27 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:27.491474 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:27.483189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:52485)
0927 22:52:27.483693 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:27.483709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:27.483866 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:27.484198 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:27.484202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:27.484602 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:27.484749 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:27.484757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.485831 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.485837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:27.486754 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:27.486765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.487496 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.487501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:27.487680 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:27.488309 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:27.488327 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:27.491259 (+ 2932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":292,"threads_started":1}
W20260927 22:52:27.491655 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:27.483225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52485 (local address 127.0.97.62:35645)
0927 22:52:27.483425 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:27.483434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:27.483800 (+ 366us) server_negotiation.cc:407] Connection header received
0927 22:52:27.483985 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:27.483990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:27.484065 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:27.484169 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:27.484890 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.485691 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.486914 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.487347 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.487487 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:27.491434 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52485: BlockingRecv error: recv got EOF from 127.0.97.1:52485 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 22:52:27 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:27 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:27.947072 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:27.938487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:35421)
0927 22:52:27.938989 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:27.939005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:27.939115 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:27.939398 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:27.939401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:27.939769 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:27.939983 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:27.939992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.941151 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.941157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:27.942213 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:27.942225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.943018 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.943029 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:27.943240 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:27.943891 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:27.943909 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:27.946738 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":310,"threads_started":1}
W20260927 22:52:27.947151 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:27.938537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35421 (local address 127.0.97.62:35645)
0927 22:52:27.938739 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:27.938745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:27.939080 (+ 335us) server_negotiation.cc:407] Connection header received
0927 22:52:27.939212 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:27.939215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:27.939282 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:27.939358 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:27.940114 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.940922 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.942380 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:27.942848 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:27.943008 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:27.946926 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35421: BlockingRecv error: recv got EOF from 127.0.97.2:35421 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 22:52:28 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:28 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:28.398900 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:28.387005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:41317)
0927 22:52:28.387524 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:28.387544 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:28.387651 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:28.387908 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:28.387911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:28.388266 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:28.388414 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:28.388421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.389498 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.389506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:28.390547 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:28.390558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.391276 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.391280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:28.391414 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:28.395648 (+ 4234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:28.395665 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:28.398542 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":305,"threads_started":1}
W20260927 22:52:28.399009 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:28.387099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41317 (local address 127.0.97.62:35645)
0927 22:52:28.387263 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:28.387269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:28.387613 (+ 344us) server_negotiation.cc:407] Connection header received
0927 22:52:28.387728 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:28.387731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:28.387791 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:28.387867 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:28.388529 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.389333 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.390670 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.391089 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.391200 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:28.398749 (+ 7549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41317: BlockingRecv error: recv got EOF from 127.0.97.3:41317 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 22:52:28 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:28 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:28.500361 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:28.492467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:36829)
0927 22:52:28.492981 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:28.492997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:28.493107 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:28.493406 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:28.493409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:28.493867 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:28.494026 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:28.494034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.494984 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.494987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:28.495873 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:28.495883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.496589 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.496593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:28.496703 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:28.497252 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:28.497270 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:28.500022 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":269,"threads_started":1}
W20260927 22:52:28.500411 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:28.492501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36829 (local address 127.0.97.62:35645)
0927 22:52:28.492728 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:28.492738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:28.493080 (+ 342us) server_negotiation.cc:407] Connection header received
0927 22:52:28.493204 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:28.493209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:28.493284 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:28.493363 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:28.494154 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.494869 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.496024 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.496416 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.496507 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:28.500191 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36829: BlockingRecv error: recv got EOF from 127.0.97.1:36829 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 22:52:28 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:28 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:28.956552 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:28.948028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:38105)
0927 22:52:28.948559 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:28.948575 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:28.948682 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:28.949026 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:28.949030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:28.949406 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:28.949562 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:28.949571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.950640 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.950645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:28.952105 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:28.952122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.952847 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.952852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:28.952962 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:28.953520 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:28.953536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:28.956225 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":293,"threads_started":1}
W20260927 22:52:28.956664 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:28.948222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38105 (local address 127.0.97.62:35645)
0927 22:52:28.948398 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:28.948402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:28.948649 (+ 247us) server_negotiation.cc:407] Connection header received
0927 22:52:28.948775 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:28.948778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:28.948865 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:28.948949 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:28.949769 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.950485 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.952245 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:28.952650 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:28.952742 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:28.956431 (+ 3689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38105: BlockingRecv error: recv got EOF from 127.0.97.2:38105 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:52:29 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:29 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:29.408068 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:29.399897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:54943)
0927 22:52:29.400381 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:29.400400 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:29.400551 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:29.400880 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:29.400887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:29.401297 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:29.401473 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:29.401482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.402494 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.402499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:29.403416 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:29.403426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.404175 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.404181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:29.404353 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:29.404974 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:29.404991 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:29.407713 (+ 2722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":247,"threads_started":1}
W20260927 22:52:29.408120 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:29.399926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54943 (local address 127.0.97.62:35645)
0927 22:52:29.400131 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:29.400139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:29.400491 (+ 352us) server_negotiation.cc:407] Connection header received
0927 22:52:29.400672 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:29.400677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:29.400741 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:29.400847 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:29.401632 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.402350 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.403571 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.403998 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.404142 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:29.407907 (+ 3765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54943: BlockingRecv error: recv got EOF from 127.0.97.3:54943 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 22:52:29 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:29 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:29.509696 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:29.501287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:49329)
0927 22:52:29.501822 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:29.501852 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:29.501960 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:29.502211 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:29.502214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:29.502544 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:29.502703 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:29.502711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.503768 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.503771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:29.504694 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:29.504704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.505397 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.505401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:29.505511 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:29.506126 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:29.506142 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:29.509286 (+ 3144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":252,"threads_started":1}
W20260927 22:52:29.509797 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:29.501487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49329 (local address 127.0.97.62:35645)
0927 22:52:29.501799 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:29.501805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:29.501906 (+ 101us) server_negotiation.cc:407] Connection header received
0927 22:52:29.502041 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:29.502044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:29.502117 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:29.502202 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:29.502826 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.503648 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.504860 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.505273 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.505372 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:29.509509 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49329: BlockingRecv error: recv got EOF from 127.0.97.1:49329 (error 108)
Metrics: {"server-negotiator.queue_time_us":179}
Sep 27 22:52:29 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:29 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:29.965253 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:29.957425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:56583)
0927 22:52:29.957953 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:29.957973 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:29.958121 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:29.958506 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:29.958511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:29.958981 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:29.959153 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:29.959162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.960229 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.960232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:29.961123 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:29.961133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.961777 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.961781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:29.961914 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:29.962484 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:29.962501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:29.965017 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":289,"threads_started":1}
W20260927 22:52:29.965389 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:29.957576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56583 (local address 127.0.97.62:35645)
0927 22:52:29.957806 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:29.957812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:29.958081 (+ 269us) server_negotiation.cc:407] Connection header received
0927 22:52:29.958262 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:29.958266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:29.958351 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:29.958449 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:29.959312 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.960105 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.961239 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:29.961667 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:29.961754 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:29.965201 (+ 3447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56583: BlockingRecv error: recv got EOF from 127.0.97.2:56583 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 22:52:30 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:30 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:30.417670 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:30.409106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.3:33859)
0927 22:52:30.409675 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:30.409695 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:30.409862 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:30.410198 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:30.410203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:30.410595 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:30.410763 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:30.410771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.411815 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.411820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:30.412743 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:30.412754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.413547 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.413553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:30.413787 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:30.414591 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:30.414616 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:30.417369 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":317,"threads_started":1}
W20260927 22:52:30.417920 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:30.409233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33859 (local address 127.0.97.62:35645)
0927 22:52:30.409505 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:30.409513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:30.409789 (+ 276us) server_negotiation.cc:407] Connection header received
0927 22:52:30.409987 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:30.409992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:30.410058 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:30.410172 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:30.410946 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.411653 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.412903 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.413388 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.413508 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:30.417657 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33859: BlockingRecv error: recv got EOF from 127.0.97.3:33859 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 22:52:30 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:30 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:30.519052 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:30.510808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.1:42349)
0927 22:52:30.511301 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:30.511316 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:30.511470 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:30.511851 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:30.511856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:30.512273 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:30.512480 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:30.512488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.513500 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.513505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:30.514484 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:30.514495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.515235 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.515240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:30.515415 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:30.516016 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:30.516033 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:30.518794 (+ 2761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":283,"threads_started":1}
W20260927 22:52:30.519200 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:30.510898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42349 (local address 127.0.97.62:35645)
0927 22:52:30.511131 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:30.511140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:30.511393 (+ 253us) server_negotiation.cc:407] Connection header received
0927 22:52:30.511624 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:30.511629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:30.511712 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:30.511816 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:30.512641 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.513354 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.514619 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.515089 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.515224 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:30.518975 (+ 3751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42349: BlockingRecv error: recv got EOF from 127.0.97.1:42349 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 22:52:30 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:52:30 dist-test-slave-df53 krb5kdc[8627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:52:30.974843 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:30.966546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35645 (local address 127.0.97.2:35413)
0927 22:52:30.967151 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:30.967168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:30.967308 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:30.967644 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:30.967649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:30.968079 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:30.968253 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:30.968261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.969271 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.969278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:30.970227 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:30.970238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.970969 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.970975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:30.971143 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:30.971790 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:30.971808 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:30.974492 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:35645: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":322,"threads_started":1}
W20260927 22:52:30.974929 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:30.966609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35413 (local address 127.0.97.62:35645)
0927 22:52:30.966842 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:30.966851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:30.967244 (+ 393us) server_negotiation.cc:407] Connection header received
0927 22:52:30.967447 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:30.967452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:30.967518 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:30.967618 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:30.968403 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.969126 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.970382 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:30.970817 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:30.970948 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:30.974678 (+ 3730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35413: BlockingRecv error: recv got EOF from 127.0.97.2:35413 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:52:31.246280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9214
I20260927 22:52:31.257318 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:31.247192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.0.97.1:39285)
0927 22:52:31.247727 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:31.247737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:31.247764 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:52:31.247818 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:31.247824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:31.248040 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:31.248203 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:31.249508 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.250564 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.251692 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.252227 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.252323 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:31.254643 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:31.254650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:31.254664 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:31.254698 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:31.256868 (+ 2170us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:52:31.257102 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58560: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":269,"threads_started":1}
W20260927 22:52:31.257448 9675 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58560: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:52:31.257525 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9214: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39285: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:52:31.257745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 9214
I20260927 22:52:31.273904 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9366
I20260927 22:52:31.285583 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:31.275003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.0.97.2:44985)
0927 22:52:31.275589 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:31.275596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:31.275626 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:52:31.275681 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:31.275687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:31.275896 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:31.276046 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:31.277403 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.278546 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.279663 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.280195 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.280288 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:31.282602 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:31.282610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:31.282623 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:31.282659 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:31.285113 (+ 2454us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:52:31.285337 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57568: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":294,"threads_started":1}
W20260927 22:52:31.285727 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9366: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:44985: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:52:31.285826 9676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57568: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:52:31.285885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 9366
I20260927 22:52:31.301429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9497
I20260927 22:52:31.312325 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:31.302272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.0.97.3:37497)
0927 22:52:31.302879 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:31.302896 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:31.302926 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:52:31.302984 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:31.302990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:31.303204 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:31.303356 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:31.304499 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.305486 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.306658 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.307217 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.307318 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:31.309506 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:31.309514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:31.309527 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:31.309562 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:31.311875 (+ 2313us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:52:31.312104 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39546: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":316,"threads_started":1}
W20260927 22:52:31.312480 9677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39546: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:52:31.312574 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9497: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:37497: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:52:31.312727 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 9497
I20260927 22:52:31.328358 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9145
I20260927 22:52:31.329509 9207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:31.441629 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 9145
2026-09-27T22:52:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20817 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T22:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:31Z Disabled control of system clock
I20260927 22:52:31.516179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33207
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:52:31.836220 9682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:31.836493 9682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:31.836567 9682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:31.843513 9682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:31.843608 9682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:31.843645 9682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:31.843691 9682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:31.843734 9682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:31.853036 9682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:31.854333 9682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:31.856076 9682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:31.867668 9687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:31.867990 9690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:31.868034 9688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:31.868315 9682 server_base.cc:1034] running on GCE node
I20260927 22:52:31.869035 9682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:31.870307 9682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:31.871565 9682 hybrid_clock.cc:648] HybridClock initialized: now 1790549551871502 us; error 52 us; skew 500 ppm
I20260927 22:52:31.871914 9682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:31.874185 9682 webserver.cc:460] Webserver started at http://127.0.97.62:42011/ using document root <none> and password file <none>
I20260927 22:52:31.874841 9682 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:31.874966 9682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:31.875334 9682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:31.878144 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "8bcd9b4091114d2eb73d7a45465776ba"
format_stamp: "Formatted at 2026-09-27 22:52:31 on dist-test-slave-df53"
server_key: "036b0528776d0ffda3467bdd481084bf"
server_key_iv: "e147690b6ae196dfd87e28f07755a790"
server_key_version: "encryptionkey@0"
I20260927 22:52:31.878875 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "8bcd9b4091114d2eb73d7a45465776ba"
format_stamp: "Formatted at 2026-09-27 22:52:31 on dist-test-slave-df53"
server_key: "036b0528776d0ffda3467bdd481084bf"
server_key_iv: "e147690b6ae196dfd87e28f07755a790"
server_key_version: "encryptionkey@0"
I20260927 22:52:31.883606 9682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 22:52:31.886898 9696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:31.888242 9682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:52:31.888417 9682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "8bcd9b4091114d2eb73d7a45465776ba"
format_stamp: "Formatted at 2026-09-27 22:52:31 on dist-test-slave-df53"
server_key: "036b0528776d0ffda3467bdd481084bf"
server_key_iv: "e147690b6ae196dfd87e28f07755a790"
server_key_version: "encryptionkey@0"
I20260927 22:52:31.888635 9682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:31.905117 9682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:31.906082 9682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:31.906395 9682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:31.924664 9682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33207
I20260927 22:52:31.924671 9747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33207 every 8 connection(s)
I20260927 22:52:31.926129 9682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:31.929731 9748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:31.936484 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 9682
I20260927 22:52:31.936678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:31.937299 388 external_mini_cluster.cc:1428] Setting key 29412f025d4725d7896c51f7623aae95
I20260927 22:52:31.939630 9748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba: Bootstrap starting.
I20260927 22:52:31.942639 9748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:31.943840 9748 log.cc:824] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:31.947026 9748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba: No bootstrap required, opened a new log
I20260927 22:52:31.951149 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:31.939355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.0.97.62:33207)
0927 22:52:31.940337 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:31.940348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:31.940388 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:52:31.942034 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:31.942056 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:31.942472 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:31.942765 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:31.943407 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.944753 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.946219 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:31.947239 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:31.947452 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:31.948424 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:31.948434 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:31.948450 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:31.949140 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:31.949248 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:31.949272 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:31.949407 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:31.949613 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 22:52:31.949932 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":346,"threads_started":1}
I20260927 22:52:31.956092 9748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd9b4091114d2eb73d7a45465776ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33207 } }
I20260927 22:52:31.956383 9748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:31.956454 9748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bcd9b4091114d2eb73d7a45465776ba, State: Initialized, Role: FOLLOWER
I20260927 22:52:31.957006 9748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd9b4091114d2eb73d7a45465776ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33207 } }
I20260927 22:52:31.957206 9748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:31.957317 9748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:31.957486 9748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:31.959798 9748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd9b4091114d2eb73d7a45465776ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33207 } }
I20260927 22:52:31.960198 9748 leader_election.cc:302] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bcd9b4091114d2eb73d7a45465776ba; no voters:
I20260927 22:52:31.960678 9748 leader_election.cc:288] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:31.960829 9753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:31.961517 9753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [term 1 LEADER]: Becoming Leader. State: Replica: 8bcd9b4091114d2eb73d7a45465776ba, State: Running, Role: LEADER
I20260927 22:52:31.962065 9753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd9b4091114d2eb73d7a45465776ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33207 } }
I20260927 22:52:31.962625 9748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:31.964982 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bcd9b4091114d2eb73d7a45465776ba. Latest consensus state: current_term: 1 leader_uuid: "8bcd9b4091114d2eb73d7a45465776ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd9b4091114d2eb73d7a45465776ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33207 } } }
I20260927 22:52:31.965366 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:31.965123 9754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bcd9b4091114d2eb73d7a45465776ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bcd9b4091114d2eb73d7a45465776ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33207 } } }
I20260927 22:52:31.966094 9754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:31.967186 9761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:31.972213 9761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:31.980363 9761 catalog_manager.cc:1347] Generated new cluster ID: ebd757fdfd3541d39ea0b051b077d201
I20260927 22:52:31.980486 9761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:32.002986 9761 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:32.004168 9761 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:32.011571 9761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bcd9b4091114d2eb73d7a45465776ba: Generated new TSK 0
I20260927 22:52:32.012406 9761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:32.021456 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33207
--builtin_ntp_servers=127.0.97.20:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 22:52:32.339955 9772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:32.340242 9772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:32.340392 9772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:32.347389 9772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:32.347478 9772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:32.347642 9772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:52:32.356362 9772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:32.357573 9772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:32.359313 9772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:32.372352 9777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:32.372543 9780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:32.372609 9778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:32.373915 9772 server_base.cc:1034] running on GCE node
I20260927 22:52:32.374492 9772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:32.375345 9772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:32.376542 9772 hybrid_clock.cc:648] HybridClock initialized: now 1790549552376485 us; error 44 us; skew 500 ppm
I20260927 22:52:32.376801 9772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:32.380003 9772 webserver.cc:460] Webserver started at http://127.0.97.1:41889/ using document root <none> and password file <none>
I20260927 22:52:32.380738 9772 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:32.380857 9772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:32.381300 9772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:32.385660 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "f1d9a3c75eb741029ae0d5ebdc1f964d"
format_stamp: "Formatted at 2026-09-27 22:52:32 on dist-test-slave-df53"
server_key: "f509b9ca2a30c51e0f6358a05a0b1c36"
server_key_iv: "ccadab77d415884ad5a8af040725b8bf"
server_key_version: "encryptionkey@0"
I20260927 22:52:32.386721 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f1d9a3c75eb741029ae0d5ebdc1f964d"
format_stamp: "Formatted at 2026-09-27 22:52:32 on dist-test-slave-df53"
server_key: "f509b9ca2a30c51e0f6358a05a0b1c36"
server_key_iv: "ccadab77d415884ad5a8af040725b8bf"
server_key_version: "encryptionkey@0"
I20260927 22:52:32.393301 9772 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260927 22:52:32.398096 9786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:32.402004 9772 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:52:32.402251 9772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "f1d9a3c75eb741029ae0d5ebdc1f964d"
format_stamp: "Formatted at 2026-09-27 22:52:32 on dist-test-slave-df53"
server_key: "f509b9ca2a30c51e0f6358a05a0b1c36"
server_key_iv: "ccadab77d415884ad5a8af040725b8bf"
server_key_version: "encryptionkey@0"
I20260927 22:52:32.402630 9772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:32.431275 9772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:32.432107 9772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:32.432384 9772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:32.433295 9772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:32.434795 9772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:32.434881 9772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:32.435030 9772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:32.435114 9772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:32.467290 9772 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43159
I20260927 22:52:32.467339 9898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43159 every 8 connection(s)
I20260927 22:52:32.468629 9772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:52:32.473245 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 9772
I20260927 22:52:32.473429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:52:32.474602 388 external_mini_cluster.cc:1428] Setting key df2393e0001aef342549728a7021361c
I20260927 22:52:32.477907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33207
--builtin_ntp_servers=127.0.97.20:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:52:32.493377 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:32.472605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35441 (local address 127.0.97.62:33207)
0927 22:52:32.474419 (+ 1814us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:32.474426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:32.474465 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:52:32.476000 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:32.476004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:32.476081 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:32.476228 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:32.485721 (+ 9493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:32.487057 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:32.489521 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:32.490283 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:32.490444 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:32.492331 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:32.492339 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:32.492347 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:32.492373 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:32.492422 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:32.492441 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:32.492532 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:32.492802 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:52:32.492905 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1673,"thread_start_us":2599,"threads_started":1}
I20260927 22:52:32.494323 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:32.472121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33207 (local address 127.0.97.1:35441)
0927 22:52:32.473409 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:32.473460 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:32.475742 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:32.476494 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:32.476507 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:32.476652 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:32.477542 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:32.477572 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:32.487254 (+ 9682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:32.487259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:32.489325 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:32.489346 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:32.490760 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:32.490769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:32.491105 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:32.492009 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:52:32.492012 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:32.492061 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:52:32.492148 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:32.492556 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:32.492566 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:32.492577 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:52:32.492791 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 22:52:32.493195 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":483,"threads_started":1}
I20260927 22:52:32.495836 9899 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33207
I20260927 22:52:32.496134 9899 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:32.496850 9899 heartbeater.cc:511] Master 127.0.97.62:33207 requested a full tablet report, sending...
I20260927 22:52:32.499100 9713 ts_manager.cc:194] Registered new tserver with Master: f1d9a3c75eb741029ae0d5ebdc1f964d (127.0.97.1:43159)
I20260927 22:52:32.501439 9713 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35441
W20260927 22:52:32.813089 9903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:32.813527 9903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:32.813688 9903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:32.820765 9903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:32.820850 9903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:32.821036 9903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:52:32.829722 9903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:32.830885 9903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:32.832545 9903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:32.845103 9908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:32.845064 9909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:32.845134 9911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:32.846323 9903 server_base.cc:1034] running on GCE node
I20260927 22:52:32.846830 9903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:32.847718 9903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:32.848919 9903 hybrid_clock.cc:648] HybridClock initialized: now 1790549552848863 us; error 51 us; skew 500 ppm
I20260927 22:52:32.849242 9903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:32.851481 9903 webserver.cc:460] Webserver started at http://127.0.97.2:43101/ using document root <none> and password file <none>
I20260927 22:52:32.852157 9903 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:32.852273 9903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:32.852592 9903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:32.855396 9903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "28461f24560b42efb44a6bc8569e01be"
format_stamp: "Formatted at 2026-09-27 22:52:32 on dist-test-slave-df53"
server_key: "10452e64f3dedda086d57e85a8bd0852"
server_key_iv: "95d8a7987ae91089c222a8ed0f5bc800"
server_key_version: "encryptionkey@0"
I20260927 22:52:32.856169 9903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "28461f24560b42efb44a6bc8569e01be"
format_stamp: "Formatted at 2026-09-27 22:52:32 on dist-test-slave-df53"
server_key: "10452e64f3dedda086d57e85a8bd0852"
server_key_iv: "95d8a7987ae91089c222a8ed0f5bc800"
server_key_version: "encryptionkey@0"
I20260927 22:52:32.860919 9903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:52:32.864508 9917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:32.865944 9903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:32.866132 9903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "28461f24560b42efb44a6bc8569e01be"
format_stamp: "Formatted at 2026-09-27 22:52:32 on dist-test-slave-df53"
server_key: "10452e64f3dedda086d57e85a8bd0852"
server_key_iv: "95d8a7987ae91089c222a8ed0f5bc800"
server_key_version: "encryptionkey@0"
I20260927 22:52:32.866362 9903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:32.895269 9903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:32.896373 9903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:32.896649 9903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:32.897946 9903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:32.899353 9903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:32.899434 9903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:32.899514 9903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:32.899572 9903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:32.931808 9903 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37443
I20260927 22:52:32.931900 10029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37443 every 8 connection(s)
I20260927 22:52:32.933068 9903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:52:32.934078 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 9903
I20260927 22:52:32.934269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:52:32.934947 388 external_mini_cluster.cc:1428] Setting key 3a6f044ed9f4f78aacff54af82972278
I20260927 22:52:32.937878 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33207
--builtin_ntp_servers=127.0.97.20:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:52:32.948513 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:32.936238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36223 (local address 127.0.97.62:33207)
0927 22:52:32.936416 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:32.936422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:32.938620 (+ 2198us) server_negotiation.cc:407] Connection header received
0927 22:52:32.941348 (+ 2728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:32.941354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:32.941424 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:32.941532 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:32.942912 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:32.943682 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:32.945861 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:32.946260 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:32.946399 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:32.947744 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:32.947749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:32.947752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:32.947768 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:32.947794 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:32.947805 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:32.947899 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:32.948130 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 22:52:32.948224 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:52:32.949463 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:32.936437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33207 (local address 127.0.97.2:36223)
0927 22:52:32.938449 (+ 2012us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:32.938507 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:32.941052 (+ 2545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:32.941819 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:32.941832 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:32.941950 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:32.942725 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:32.942750 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:32.943832 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:32.943840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:32.945720 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:32.945731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:32.946477 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:32.946483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:32.946717 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:32.947436 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:52:32.947439 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:32.947475 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:52:32.947551 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:32.947898 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:32.947909 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:32.947919 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:52:32.948108 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 22:52:32.948467 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1431,"thread_start_us":1222,"threads_started":1}
I20260927 22:52:32.950855 10030 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33207
I20260927 22:52:32.951210 10030 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:32.952068 10030 heartbeater.cc:511] Master 127.0.97.62:33207 requested a full tablet report, sending...
I20260927 22:52:32.953526 9713 ts_manager.cc:194] Registered new tserver with Master: 28461f24560b42efb44a6bc8569e01be (127.0.97.2:37443)
I20260927 22:52:32.954798 9713 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:36223
W20260927 22:52:33.256008 10033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:33.256285 10033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:33.256364 10033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:33.263566 10033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:33.263656 10033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:33.263818 10033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:52:33.273068 10033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:33.274330 10033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:33.275967 10033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:33.288769 10038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:33.288815 10039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:33.288836 10041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:33.289674 10033 server_base.cc:1034] running on GCE node
I20260927 22:52:33.290274 10033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:33.291261 10033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:33.292482 10033 hybrid_clock.cc:648] HybridClock initialized: now 1790549553292446 us; error 47 us; skew 500 ppm
I20260927 22:52:33.292786 10033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:33.295038 10033 webserver.cc:460] Webserver started at http://127.0.97.3:41653/ using document root <none> and password file <none>
I20260927 22:52:33.295682 10033 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:33.295816 10033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:33.296161 10033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:33.298954 10033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "20350f7a6c174659812a02ebe1fe1803"
format_stamp: "Formatted at 2026-09-27 22:52:33 on dist-test-slave-df53"
server_key: "df0150349aeec203d157499839a0385b"
server_key_iv: "23fe390082ac5f08cc6bf62ad5b0cfe4"
server_key_version: "encryptionkey@0"
I20260927 22:52:33.299741 10033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "20350f7a6c174659812a02ebe1fe1803"
format_stamp: "Formatted at 2026-09-27 22:52:33 on dist-test-slave-df53"
server_key: "df0150349aeec203d157499839a0385b"
server_key_iv: "23fe390082ac5f08cc6bf62ad5b0cfe4"
server_key_version: "encryptionkey@0"
I20260927 22:52:33.304538 10033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:52:33.307757 10047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:33.308986 10033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:52:33.309175 10033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "20350f7a6c174659812a02ebe1fe1803"
format_stamp: "Formatted at 2026-09-27 22:52:33 on dist-test-slave-df53"
server_key: "df0150349aeec203d157499839a0385b"
server_key_iv: "23fe390082ac5f08cc6bf62ad5b0cfe4"
server_key_version: "encryptionkey@0"
I20260927 22:52:33.309382 10033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:33.325811 10033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:33.326594 10033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:33.326871 10033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:33.327775 10033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:33.329145 10033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:33.329269 10033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:33.329421 10033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:33.329512 10033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:33.362020 10033 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45071
I20260927 22:52:33.362046 10159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45071 every 8 connection(s)
I20260927 22:52:33.363317 10033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:52:33.371274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 10033
I20260927 22:52:33.371428 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:52:33.371857 388 external_mini_cluster.cc:1428] Setting key f52b7a1eb0c4e829fb7d63b2138a1271
I20260927 22:52:33.376302 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:33.365911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52531 (local address 127.0.97.62:33207)
0927 22:52:33.366086 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:33.366090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:33.367510 (+ 1420us) server_negotiation.cc:407] Connection header received
0927 22:52:33.369243 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:33.369249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:33.369343 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:33.369455 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:33.370513 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.371450 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.373387 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.373964 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.374061 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:33.375607 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:33.375612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:33.375616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:33.375632 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:33.375661 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:33.375672 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:33.375747 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:33.375975 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 22:52:33.376075 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:52:33.377185 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:33.366389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33207 (local address 127.0.97.3:52531)
0927 22:52:33.367349 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:33.367384 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:33.368987 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:33.369731 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:33.369739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:33.369823 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:33.370349 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:33.370364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.371610 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.371614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:33.373237 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:33.373249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.374247 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.374254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:33.374536 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:33.375330 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:52:33.375333 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:33.375370 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:52:33.375439 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:33.375763 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:33.375769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:33.375775 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:52:33.375910 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:52:33.376186 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":337,"threads_started":1}
I20260927 22:52:33.378491 10160 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33207
I20260927 22:52:33.378724 10160 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:33.379400 10160 heartbeater.cc:511] Master 127.0.97.62:33207 requested a full tablet report, sending...
I20260927 22:52:33.380625 9713 ts_manager.cc:194] Registered new tserver with Master: 20350f7a6c174659812a02ebe1fe1803 (127.0.97.3:45071)
I20260927 22:52:33.381421 9713 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52531
I20260927 22:52:33.387086 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:52:33.393126 9902 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:33.391453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.0.97.62:33207)
0927 22:52:33.391657 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:33.391661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:33.392172 (+ 511us) server_negotiation.cc:407] Connection header received
0927 22:52:33.392423 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:33.392426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:33.392495 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:33.392578 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:33.392880 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56088: BlockingRecv error: recv got EOF from 127.0.0.1:56088 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:52:33.393682 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9772
I20260927 22:52:33.406241 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:33.394727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.0.97.1:43159)
0927 22:52:33.395251 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:33.395257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:33.395281 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:52:33.395331 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:33.395336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:33.395518 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:33.395640 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:33.396016 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.396999 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.398279 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.398807 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.398909 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:33.404848 (+ 5939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:33.404856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:33.404868 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:33.405464 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:33.405510 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:33.405529 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:33.405692 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:33.405834 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 22:52:33.405942 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
I20260927 22:52:33.407450 9894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:33.488165 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 9772
I20260927 22:52:33.501454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9903
I20260927 22:52:33.508769 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:33.502376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.0.97.2:37443)
0927 22:52:33.502919 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:33.502927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:33.502952 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:52:33.503015 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:33.503021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:33.503232 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:33.503365 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:33.503688 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.504647 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.505865 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.506409 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.506508 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:33.507439 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:33.507447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:33.507461 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:33.508020 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:33.508061 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:33.508096 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:33.508188 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:33.508360 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 22:52:33.508463 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":296,"threads_started":1}
I20260927 22:52:33.510054 10025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:33.594713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 9903
I20260927 22:52:33.610497 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 10033
I20260927 22:52:33.618080 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:33.611349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.0.97.3:45071)
0927 22:52:33.611952 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:33.611965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:33.611995 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:52:33.612053 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:33.612060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:33.612274 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:33.612399 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:33.612789 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.613946 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.615236 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:33.615770 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:33.615861 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:33.616785 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:33.616792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:33.616805 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:33.617384 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:33.617429 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:33.617446 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:33.617538 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:33.617695 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:52:33.617793 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":300,"threads_started":1}
I20260927 22:52:33.619259 10155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:33.700151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 10033
I20260927 22:52:33.715399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 9682
I20260927 22:52:33.716550 9743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:33.815964 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 9682
2026-09-27T22:52:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2368 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T22:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:33Z Disabled control of system clock
I20260927 22:52:33.883384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45093
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:52:34.187896 10179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:34.188212 10179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:34.188371 10179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:34.195187 10179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:34.195318 10179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:34.195406 10179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:34.195488 10179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:34.195551 10179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:34.204035 10179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:34.205188 10179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:34.206912 10179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:34.218525 10184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:34.218468 10187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:34.218554 10185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:34.219586 10179 server_base.cc:1034] running on GCE node
I20260927 22:52:34.220279 10179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:34.221542 10179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:34.222796 10179 hybrid_clock.cc:648] HybridClock initialized: now 1790549554222749 us; error 52 us; skew 500 ppm
I20260927 22:52:34.223116 10179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:34.225256 10179 webserver.cc:460] Webserver started at http://127.0.97.62:38075/ using document root <none> and password file <none>
I20260927 22:52:34.225972 10179 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:34.226106 10179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:34.226416 10179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:34.229118 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "f503e82a20554f81a36f7748417d2b70"
format_stamp: "Formatted at 2026-09-27 22:52:34 on dist-test-slave-df53"
server_key: "d25fb0b863b37ebed4c2ed81cfd9c48a"
server_key_iv: "01da0173f76eafe53cc624208de7aacb"
server_key_version: "encryptionkey@0"
I20260927 22:52:34.229931 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "f503e82a20554f81a36f7748417d2b70"
format_stamp: "Formatted at 2026-09-27 22:52:34 on dist-test-slave-df53"
server_key: "d25fb0b863b37ebed4c2ed81cfd9c48a"
server_key_iv: "01da0173f76eafe53cc624208de7aacb"
server_key_version: "encryptionkey@0"
I20260927 22:52:34.234707 10179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:52:34.237753 10193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:34.238952 10179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:52:34.239120 10179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "f503e82a20554f81a36f7748417d2b70"
format_stamp: "Formatted at 2026-09-27 22:52:34 on dist-test-slave-df53"
server_key: "d25fb0b863b37ebed4c2ed81cfd9c48a"
server_key_iv: "01da0173f76eafe53cc624208de7aacb"
server_key_version: "encryptionkey@0"
I20260927 22:52:34.239321 10179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:34.244489 10179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:34.245208 10179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:34.245494 10179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:34.263185 10179 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45093
I20260927 22:52:34.263224 10244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45093 every 8 connection(s)
I20260927 22:52:34.264379 10179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:34.267804 10245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:34.269946 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 10179
I20260927 22:52:34.270162 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:34.270759 388 external_mini_cluster.cc:1428] Setting key f8759a9249995494fee8c7abe5f3eea0
I20260927 22:52:34.279798 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:34.272690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.0.97.62:45093)
0927 22:52:34.273731 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:34.273743 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:34.273790 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:52:34.276009 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:34.276036 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:34.276568 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:34.276986 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:34.277983 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:34.277990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:34.278006 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:34.278021 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:34.278091 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:34.278120 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:34.278336 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:34.278474 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 22:52:34.278715 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":308,"threads_started":1}
I20260927 22:52:34.280443 10245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70: Bootstrap starting.
I20260927 22:52:34.283339 10245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:34.284250 10245 log.cc:824] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:34.286756 10245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70: No bootstrap required, opened a new log
I20260927 22:52:34.293314 10245 raft_consensus.cc:348] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503e82a20554f81a36f7748417d2b70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45093 } }
I20260927 22:52:34.293658 10245 raft_consensus.cc:374] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:34.293773 10245 raft_consensus.cc:729] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f503e82a20554f81a36f7748417d2b70, State: Initialized, Role: FOLLOWER
I20260927 22:52:34.294351 10245 consensus_queue.cc:260] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503e82a20554f81a36f7748417d2b70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45093 } }
I20260927 22:52:34.294570 10245 raft_consensus.cc:388] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:34.294690 10245 raft_consensus.cc:482] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:34.294852 10245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:34.296856 10245 raft_consensus.cc:504] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503e82a20554f81a36f7748417d2b70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45093 } }
I20260927 22:52:34.297319 10245 leader_election.cc:302] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f503e82a20554f81a36f7748417d2b70; no voters:
I20260927 22:52:34.297885 10245 leader_election.cc:288] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:34.298002 10250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:34.298735 10250 raft_consensus.cc:686] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [term 1 LEADER]: Becoming Leader. State: Replica: f503e82a20554f81a36f7748417d2b70, State: Running, Role: LEADER
I20260927 22:52:34.299206 10250 consensus_queue.cc:237] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503e82a20554f81a36f7748417d2b70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45093 } }
I20260927 22:52:34.299852 10245 sys_catalog.cc:564] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:34.301802 10252 sys_catalog.cc:455] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f503e82a20554f81a36f7748417d2b70. Latest consensus state: current_term: 1 leader_uuid: "f503e82a20554f81a36f7748417d2b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503e82a20554f81a36f7748417d2b70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45093 } } }
I20260927 22:52:34.302202 10252 sys_catalog.cc:458] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:34.302121 10251 sys_catalog.cc:455] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f503e82a20554f81a36f7748417d2b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503e82a20554f81a36f7748417d2b70" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45093 } } }
I20260927 22:52:34.302448 10251 sys_catalog.cc:458] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:34.304392 10258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:34.310302 10258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:34.318081 10258 catalog_manager.cc:1347] Generated new cluster ID: 18c989b3554847ecaf304f7ab3774fdf
I20260927 22:52:34.318178 10258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:34.328269 10258 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:34.328653 10258 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:34.345695 10258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f503e82a20554f81a36f7748417d2b70: Generated new TSK 0
I20260927 22:52:34.346485 10258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:52:34.359382 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45093
--builtin_ntp_servers=127.0.97.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:52:34.687891 10269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:34.688163 10269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:34.688249 10269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:34.695405 10269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:34.695500 10269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:34.695611 10269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:52:34.704574 10269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:34.705787 10269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:34.707535 10269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:34.719745 10274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:34.719795 10275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:34.719866 10277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:34.720831 10269 server_base.cc:1034] running on GCE node
I20260927 22:52:34.721391 10269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:34.722319 10269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:34.723564 10269 hybrid_clock.cc:648] HybridClock initialized: now 1790549554723519 us; error 53 us; skew 500 ppm
I20260927 22:52:34.723891 10269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:34.726110 10269 webserver.cc:460] Webserver started at http://127.0.97.1:42891/ using document root <none> and password file <none>
I20260927 22:52:34.726770 10269 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:34.726897 10269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:34.727213 10269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:34.729992 10269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "c3d6ac140e414131b4feff1f43a892f7"
format_stamp: "Formatted at 2026-09-27 22:52:34 on dist-test-slave-df53"
server_key: "9af9d46dddcd9d422eaf72cbf1b379c7"
server_key_iv: "8caea68abfa5306cd55ad301f35b8361"
server_key_version: "encryptionkey@0"
I20260927 22:52:34.730749 10269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c3d6ac140e414131b4feff1f43a892f7"
format_stamp: "Formatted at 2026-09-27 22:52:34 on dist-test-slave-df53"
server_key: "9af9d46dddcd9d422eaf72cbf1b379c7"
server_key_iv: "8caea68abfa5306cd55ad301f35b8361"
server_key_version: "encryptionkey@0"
I20260927 22:52:34.735502 10269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:52:34.738847 10283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:34.740139 10269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:52:34.740301 10269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "c3d6ac140e414131b4feff1f43a892f7"
format_stamp: "Formatted at 2026-09-27 22:52:34 on dist-test-slave-df53"
server_key: "9af9d46dddcd9d422eaf72cbf1b379c7"
server_key_iv: "8caea68abfa5306cd55ad301f35b8361"
server_key_version: "encryptionkey@0"
I20260927 22:52:34.740525 10269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:34.745929 10269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:34.746675 10269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:34.746945 10269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:34.747797 10269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:34.749221 10269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:34.749338 10269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:34.749462 10269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:34.749558 10269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:34.782644 10269 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40485
I20260927 22:52:34.782737 10395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40485 every 8 connection(s)
I20260927 22:52:34.784029 10269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:52:34.792418 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 10269
I20260927 22:52:34.792654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:52:34.792793 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:34.786760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42265 (local address 127.0.97.62:45093)
0927 22:52:34.787253 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:34.787258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:34.788422 (+ 1164us) server_negotiation.cc:407] Connection header received
0927 22:52:34.790327 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:34.790333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:34.790401 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:34.790535 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:34.791878 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:34.791885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:34.791889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:34.791909 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:34.791962 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:34.791979 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:34.792116 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:34.792358 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 22:52:34.792426 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":311,"threads_started":1}
I20260927 22:52:34.793398 388 external_mini_cluster.cc:1428] Setting key b0d3fe47f7e7b768048558e1db9953ed
I20260927 22:52:34.793733 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:34.787295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45093 (local address 127.0.97.1:42265)
0927 22:52:34.788262 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:34.788298 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:34.790045 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:34.790777 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:34.790785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:34.790878 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:34.791583 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:52:34.791585 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:34.791623 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:52:34.791705 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:34.792141 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:34.792149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:34.792156 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:52:34.792379 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 22:52:34.792628 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":304,"threads_started":1}
I20260927 22:52:34.795233 10396 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45093
I20260927 22:52:34.795509 10396 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:34.796046 10396 heartbeater.cc:511] Master 127.0.97.62:45093 requested a full tablet report, sending...
I20260927 22:52:34.796324 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45093
--builtin_ntp_servers=127.0.97.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:52:34.798112 10210 ts_manager.cc:194] Registered new tserver with Master: c3d6ac140e414131b4feff1f43a892f7 (127.0.97.1:40485)
W20260927 22:52:35.104662 10400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:35.104918 10400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:35.105070 10400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:35.112280 10400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:35.112373 10400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:35.112581 10400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:52:35.121359 10400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:35.122602 10400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:35.124305 10400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:35.136739 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:35.136812 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:35.136786 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:35.137857 10400 server_base.cc:1034] running on GCE node
I20260927 22:52:35.138408 10400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:35.139269 10400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:35.140496 10400 hybrid_clock.cc:648] HybridClock initialized: now 1790549555140461 us; error 46 us; skew 500 ppm
I20260927 22:52:35.140796 10400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:35.143193 10400 webserver.cc:460] Webserver started at http://127.0.97.2:35635/ using document root <none> and password file <none>
I20260927 22:52:35.143939 10400 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:35.144021 10400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:35.144367 10400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:35.147279 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "aa0a8193f28f49458106255d2530ca3a"
format_stamp: "Formatted at 2026-09-27 22:52:35 on dist-test-slave-df53"
server_key: "0683b47d77b708c17a0411c2dfbd1243"
server_key_iv: "6475be4fcd9a6ce04feac2d1b754bb83"
server_key_version: "encryptionkey@0"
I20260927 22:52:35.148002 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "aa0a8193f28f49458106255d2530ca3a"
format_stamp: "Formatted at 2026-09-27 22:52:35 on dist-test-slave-df53"
server_key: "0683b47d77b708c17a0411c2dfbd1243"
server_key_iv: "6475be4fcd9a6ce04feac2d1b754bb83"
server_key_version: "encryptionkey@0"
I20260927 22:52:35.152596 10400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 22:52:35.155791 10414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:35.156946 10400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 22:52:35.157105 10400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "aa0a8193f28f49458106255d2530ca3a"
format_stamp: "Formatted at 2026-09-27 22:52:35 on dist-test-slave-df53"
server_key: "0683b47d77b708c17a0411c2dfbd1243"
server_key_iv: "6475be4fcd9a6ce04feac2d1b754bb83"
server_key_version: "encryptionkey@0"
I20260927 22:52:35.157294 10400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:35.162482 10400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:35.163264 10400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:35.163545 10400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:35.164410 10400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:35.165735 10400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:35.165845 10400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:35.165972 10400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:35.166087 10400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:35.197547 10400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45159
I20260927 22:52:35.197650 10526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45159 every 8 connection(s)
I20260927 22:52:35.198844 10400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:52:35.206823 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:35.201494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60121 (local address 127.0.97.62:45093)
0927 22:52:35.201712 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:35.201717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:35.203069 (+ 1352us) server_negotiation.cc:407] Connection header received
0927 22:52:35.204660 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:35.204664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:35.204723 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:35.204817 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:35.206049 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:35.206053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:35.206056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:35.206070 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:35.206130 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:35.206148 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:35.206263 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:35.206425 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 22:52:35.206471 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:52:35.206861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 10400
I20260927 22:52:35.207052 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:52:35.207470 388 external_mini_cluster.cc:1428] Setting key 2ca99e575d9d22eb502e3be8f5973869
I20260927 22:52:35.207705 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:35.202046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45093 (local address 127.0.97.2:60121)
0927 22:52:35.202889 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:35.202933 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:35.204413 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:35.205004 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:35.205012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:35.205097 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:35.205784 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:52:35.205787 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:35.205822 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:52:35.205901 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:35.206252 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:35.206260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:35.206266 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:52:35.206472 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 22:52:35.206715 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":244,"threads_started":1}
I20260927 22:52:35.209213 10527 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45093
I20260927 22:52:35.209506 10527 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:35.210080 10527 heartbeater.cc:511] Master 127.0.97.62:45093 requested a full tablet report, sending...
I20260927 22:52:35.210369 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45093
--builtin_ntp_servers=127.0.97.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:52:35.211443 10210 ts_manager.cc:194] Registered new tserver with Master: aa0a8193f28f49458106255d2530ca3a (127.0.97.2:45159)
W20260927 22:52:35.524988 10530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:35.525302 10530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:35.525463 10530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:35.532557 10530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:35.532686 10530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:35.532851 10530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:52:35.541679 10530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:35.542917 10530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:35.544760 10530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:35.557303 10535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:35.557490 10536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:35.557557 10538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:35.558162 10530 server_base.cc:1034] running on GCE node
I20260927 22:52:35.558720 10530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:35.559571 10530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:35.560809 10530 hybrid_clock.cc:648] HybridClock initialized: now 1790549555560746 us; error 46 us; skew 500 ppm
I20260927 22:52:35.561178 10530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:52:35.563459 10530 webserver.cc:460] Webserver started at http://127.0.97.3:44501/ using document root <none> and password file <none>
I20260927 22:52:35.564123 10530 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:35.564253 10530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:35.564567 10530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:35.567355 10530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "981e8050829c4c1a85979cd8901a449f"
format_stamp: "Formatted at 2026-09-27 22:52:35 on dist-test-slave-df53"
server_key: "ed14e425027bda2904fa26e1fe2351ba"
server_key_iv: "627c4c73be664c334a66d7e9e3a9e779"
server_key_version: "encryptionkey@0"
I20260927 22:52:35.568120 10530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "981e8050829c4c1a85979cd8901a449f"
format_stamp: "Formatted at 2026-09-27 22:52:35 on dist-test-slave-df53"
server_key: "ed14e425027bda2904fa26e1fe2351ba"
server_key_iv: "627c4c73be664c334a66d7e9e3a9e779"
server_key_version: "encryptionkey@0"
I20260927 22:52:35.574141 10530 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 22:52:35.578606 10544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:35.580133 10530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:52:35.580308 10530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "981e8050829c4c1a85979cd8901a449f"
format_stamp: "Formatted at 2026-09-27 22:52:35 on dist-test-slave-df53"
server_key: "ed14e425027bda2904fa26e1fe2351ba"
server_key_iv: "627c4c73be664c334a66d7e9e3a9e779"
server_key_version: "encryptionkey@0"
I20260927 22:52:35.580613 10530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:35.586071 10530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:35.586858 10530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:35.587211 10530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:35.588354 10530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:35.589900 10530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:35.590016 10530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:35.590128 10530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:35.590217 10530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:35.621227 10530 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40381
I20260927 22:52:35.621274 10656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40381 every 8 connection(s)
I20260927 22:52:35.622576 10530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:52:35.630093 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:35.624976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45707 (local address 127.0.97.62:45093)
0927 22:52:35.625159 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:35.625164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:35.626542 (+ 1378us) server_negotiation.cc:407] Connection header received
0927 22:52:35.628161 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:35.628166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:35.628218 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:35.628315 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:35.629482 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:35.629486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:35.629490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:35.629504 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:35.629532 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:35.629543 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:35.629698 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:35.629853 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 22:52:35.629898 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:52:35.630663 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 10530
I20260927 22:52:35.630853 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:52:35.631199 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:35.625471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45093 (local address 127.0.97.3:45707)
0927 22:52:35.626380 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:35.626415 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:35.627882 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:35.628507 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:35.628517 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:35.628604 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:35.629206 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:52:35.629208 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:35.629249 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:52:35.629327 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:35.629682 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:35.629689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:35.629695 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:52:35.629901 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 22:52:35.630080 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":310,"threads_started":1}
I20260927 22:52:35.631284 388 external_mini_cluster.cc:1428] Setting key c73ece0f2851f0032ed00ccbd4097b90
I20260927 22:52:35.632532 10657 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45093
I20260927 22:52:35.632861 10657 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:35.633441 10657 heartbeater.cc:511] Master 127.0.97.62:45093 requested a full tablet report, sending...
I20260927 22:52:35.634864 10209 ts_manager.cc:194] Registered new tserver with Master: 981e8050829c4c1a85979cd8901a449f (127.0.97.3:40381)
I20260927 22:52:35.646132 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:52:35.652071 10398 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:52:35.650514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.0.97.62:45093)
0927 22:52:35.650657 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:35.650661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:35.651139 (+ 478us) server_negotiation.cc:407] Connection header received
0927 22:52:35.651338 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:35.651341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:35.651386 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:35.651484 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:35.651811 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52678: BlockingRecv error: recv got EOF from 127.0.0.1:52678 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:52:35.652829 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 10269
I20260927 22:52:35.656493 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:35.653786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.0.97.1:40485)
0927 22:52:35.654381 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:35.654387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:35.654411 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:52:35.654469 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:35.654479 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:35.654770 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:35.654954 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:35.655813 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:35.655819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:35.655830 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:35.655847 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:35.655887 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:35.655900 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:35.656039 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:35.656102 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 22:52:35.656255 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":292,"threads_started":1}
I20260927 22:52:35.657648 10391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:35.737177 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 10269
I20260927 22:52:35.753374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 10400
I20260927 22:52:35.756870 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:35.754375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.0.97.2:45159)
0927 22:52:35.754852 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:35.754859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:35.754884 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:52:35.754942 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:35.754948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:35.755152 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:35.755311 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:35.756193 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:35.756199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:35.756209 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:35.756234 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:35.756271 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:35.756284 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:35.756384 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:35.756472 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 22:52:35.756586 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":230,"threads_started":1}
I20260927 22:52:35.758214 10522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:35.841058 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 10400
I20260927 22:52:35.856493 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 10530
I20260927 22:52:35.860461 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:35.857471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.0.97.3:40381)
0927 22:52:35.858106 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:35.858113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:35.858142 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:52:35.858208 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:35.858214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:35.858429 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:35.858603 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:35.859728 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:35.859734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:35.859746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:52:35.859761 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:35.859799 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:52:35.859811 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:35.859991 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:35.860065 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:52:35.860171 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":348,"threads_started":1}
I20260927 22:52:35.861747 10652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:35.941802 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 10530
I20260927 22:52:35.957463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 10179
I20260927 22:52:35.958621 10240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:36.056712 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 10179
2026-09-27T22:52:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2240 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:36 dist-test-slave-df53 krb5kdc[10675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:36 dist-test-slave-df53 krb5kdc[10675](info): set up 2 sockets
Sep 27 22:52:36 dist-test-slave-df53 krb5kdc[10675](info): commencing operation
krb5kdc: starting...
W20260927 22:52:37.150447 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:37 dist-test-slave-df53 krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:52:37.359581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36015
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:37.671007 10688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:37.671343 10688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:37.671506 10688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:37.678382 10688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:37.678526 10688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:37.678632 10688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:37.678736 10688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:37.678843 10688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:37.688830 10688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:37.690248 10688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:37.691491 10688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:37.702782 10696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:37.702973 10693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:37.702920 10694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:37.703706 10688 server_base.cc:1034] running on GCE node
I20260927 22:52:37.704383 10688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:37.705679 10688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:37.706889 10688 hybrid_clock.cc:648] HybridClock initialized: now 1790549557706845 us; error 40 us; skew 500 ppm
I20260927 22:52:37.707239 10688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:37 dist-test-slave-df53 krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:37.710403 10688 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:52:37.711826 10688 webserver.cc:460] Webserver started at http://127.0.97.62:40033/ using document root <none> and password file <none>
I20260927 22:52:37.712476 10688 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:37.712602 10688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:37.712900 10688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:37.715739 10688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "555e5af87fbc4586afe97bec4d933831"
format_stamp: "Formatted at 2026-09-27 22:52:37 on dist-test-slave-df53"
server_key: "eac9c71d05ddc6229a7dc7022a62808f"
server_key_iv: "301686026788804bb10fd64fab46c000"
server_key_version: "encryptionkey@0"
I20260927 22:52:37.716468 10688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "555e5af87fbc4586afe97bec4d933831"
format_stamp: "Formatted at 2026-09-27 22:52:37 on dist-test-slave-df53"
server_key: "eac9c71d05ddc6229a7dc7022a62808f"
server_key_iv: "301686026788804bb10fd64fab46c000"
server_key_version: "encryptionkey@0"
I20260927 22:52:37.721866 10688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:52:37.724956 10703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:37.726250 10688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:37.726439 10688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "555e5af87fbc4586afe97bec4d933831"
format_stamp: "Formatted at 2026-09-27 22:52:37 on dist-test-slave-df53"
server_key: "eac9c71d05ddc6229a7dc7022a62808f"
server_key_iv: "301686026788804bb10fd64fab46c000"
server_key_version: "encryptionkey@0"
I20260927 22:52:37.726624 10688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:37.739187 10688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:37.743391 10688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:37.743667 10688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:37.761534 10688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36015
I20260927 22:52:37.761565 10754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36015 every 8 connection(s)
I20260927 22:52:37.762856 10688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:37.766324 10755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:37.771395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 10688
I20260927 22:52:37.771628 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:52:37.772245 388 external_mini_cluster.cc:1428] Setting key c0e3ed372ff7ec08b057ed280048aaa5
I20260927 22:52:37.778244 10755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831: Bootstrap starting.
I20260927 22:52:37.781809 10755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:37.782759 10755 log.cc:824] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:37.785445 10755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831: No bootstrap required, opened a new log
Sep 27 22:52:37 dist-test-slave-df53 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:37.795351 10755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e5af87fbc4586afe97bec4d933831" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36015 } }
I20260927 22:52:37.795773 10755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:37.795894 10755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555e5af87fbc4586afe97bec4d933831, State: Initialized, Role: FOLLOWER
I20260927 22:52:37.796422 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:37.774240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.0.97.62:36015)
0927 22:52:37.775126 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:37.775145 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:37.775198 (+ 53us) server_negotiation.cc:407] Connection header received
0927 22:52:37.777474 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:37.777499 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:37.778006 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:37.778436 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:37.779762 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.781759 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.783095 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:37.783950 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:37.784140 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:37.788028 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:37.788039 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:37.788057 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:37.788098 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:37.792077 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.792751 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.792759 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.792767 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.792978 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:37.793342 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:37.793347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:37.793350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:37.794068 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:37.794351 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:37.794705 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 22:52:37.795138 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":404,"threads_started":1}
I20260927 22:52:37.796566 10755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e5af87fbc4586afe97bec4d933831" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36015 } }
I20260927 22:52:37.796837 10755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:37.797014 10755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:37.797195 10755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:37.799644 10755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e5af87fbc4586afe97bec4d933831" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36015 } }
I20260927 22:52:37.800132 10755 leader_election.cc:302] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 555e5af87fbc4586afe97bec4d933831; no voters:
I20260927 22:52:37.800745 10755 leader_election.cc:288] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:37.800887 10760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:37.801563 10760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [term 1 LEADER]: Becoming Leader. State: Replica: 555e5af87fbc4586afe97bec4d933831, State: Running, Role: LEADER
I20260927 22:52:37.802294 10760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e5af87fbc4586afe97bec4d933831" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36015 } }
I20260927 22:52:37.802959 10755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:37.805171 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 555e5af87fbc4586afe97bec4d933831. Latest consensus state: current_term: 1 leader_uuid: "555e5af87fbc4586afe97bec4d933831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e5af87fbc4586afe97bec4d933831" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36015 } } }
I20260927 22:52:37.805569 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:37.806424 10761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "555e5af87fbc4586afe97bec4d933831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555e5af87fbc4586afe97bec4d933831" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36015 } } }
I20260927 22:52:37.806792 10761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:37.808657 10767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:37.813010 10767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:37.820945 10767 catalog_manager.cc:1347] Generated new cluster ID: 16370d9d7e404ca1b432fd046084554a
I20260927 22:52:37.821041 10767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:37.842255 10767 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:37.843494 10767 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:37.853556 10767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831: Generated new TSK 0
I20260927 22:52:37.854363 10767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:52:37.908965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36015
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:38.223181 10781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:38.223434 10781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:38.223606 10781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:38.230629 10781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:38.230746 10781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:38.230937 10781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:52:38.240085 10781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:38.241273 10781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:38.242522 10781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:38.254981 10786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:38.254894 10787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:38.254927 10789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:38.255752 10781 server_base.cc:1034] running on GCE node
I20260927 22:52:38.256285 10781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:38.257072 10781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:38.258301 10781 hybrid_clock.cc:648] HybridClock initialized: now 1790549558258234 us; error 55 us; skew 500 ppm
I20260927 22:52:38.258563 10781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:38 dist-test-slave-df53 krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:38.261521 10781 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:52:38.262897 10781 webserver.cc:460] Webserver started at http://127.0.97.1:46797/ using document root <none> and password file <none>
I20260927 22:52:38.263540 10781 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:38.263664 10781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:38.263967 10781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:38.266716 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "b173ed5d612e462d9047af2895595bde"
format_stamp: "Formatted at 2026-09-27 22:52:38 on dist-test-slave-df53"
server_key: "1e8ead14a1579946a6380cabf4d9fc01"
server_key_iv: "37316cd8907349182774c66bb06ca579"
server_key_version: "encryptionkey@0"
I20260927 22:52:38.267457 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b173ed5d612e462d9047af2895595bde"
format_stamp: "Formatted at 2026-09-27 22:52:38 on dist-test-slave-df53"
server_key: "1e8ead14a1579946a6380cabf4d9fc01"
server_key_iv: "37316cd8907349182774c66bb06ca579"
server_key_version: "encryptionkey@0"
I20260927 22:52:38.272212 10781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:52:38.275363 10796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:38.276594 10781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:52:38.276773 10781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "b173ed5d612e462d9047af2895595bde"
format_stamp: "Formatted at 2026-09-27 22:52:38 on dist-test-slave-df53"
server_key: "1e8ead14a1579946a6380cabf4d9fc01"
server_key_iv: "37316cd8907349182774c66bb06ca579"
server_key_version: "encryptionkey@0"
I20260927 22:52:38.276965 10781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:38.306886 10781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:38.310977 10781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:38.311281 10781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:38.312191 10781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:38.313472 10781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:38.313578 10781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:38.313743 10781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:38.313830 10781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:38.346513 10781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35213
I20260927 22:52:38.346601 10908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35213 every 8 connection(s)
I20260927 22:52:38.348290 10781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:52:38.349618 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 10781
I20260927 22:52:38.349869 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:52:38.350590 388 external_mini_cluster.cc:1428] Setting key 34a4873e8b7db36c8c122681def3d62b
Sep 27 22:52:38 dist-test-slave-df53 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:38.371387 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.351091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51279 (local address 127.0.97.62:36015)
0927 22:52:38.351581 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.351587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.353694 (+ 2107us) server_negotiation.cc:407] Connection header received
0927 22:52:38.355554 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.355562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.355677 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.355785 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.357755 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.358473 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.360971 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.361365 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.361492 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.364810 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:38.364820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:38.364828 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:38.364870 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:38.368399 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:38.369084 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:38.369092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:38.369098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:38.369238 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:38.369666 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:38.369673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:38.369679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:38.370057 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:38.370208 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.370819 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.371026 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260927 22:52:38.372283 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.351724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36015 (local address 127.0.97.1:51279)
0927 22:52:38.352539 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:38.352584 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:38.355236 (+ 2652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:38.356045 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:38.356058 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:38.356792 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:38.357561 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:38.357611 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.358630 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.358638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:38.360812 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:38.360834 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.361618 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.361626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:38.361858 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.362523 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:38.362544 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:38.364590 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:38.368531 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:38.368538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:38.368550 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:38.368945 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:38.369347 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:38.369350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:38.369352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:38.369514 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:38.370177 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:38.370183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:38.370459 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 22:52:38.370710 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 22:52:38.371193 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":306,"threads_started":1}
I20260927 22:52:38.373872 10909 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36015
I20260927 22:52:38.374208 10909 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:52:38.374917 10909 heartbeater.cc:511] Master 127.0.97.62:36015 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:52:38.377278 10720 ts_manager.cc:194] Registered new tserver with Master: b173ed5d612e462d9047af2895595bde (127.0.97.1:35213)
I20260927 22:52:38.378719 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:52:38.404752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36015
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:38.720345 10915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:38.720600 10915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:38.720691 10915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:38.727784 10915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:38.727880 10915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:38.727989 10915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:52:38.737210 10915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:38.738371 10915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:38.739563 10915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:38.752151 10923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:38.752287 10921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:38.752682 10920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:38.753402 10915 server_base.cc:1034] running on GCE node
I20260927 22:52:38.754038 10915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:38.754890 10915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:38.756126 10915 hybrid_clock.cc:648] HybridClock initialized: now 1790549558756090 us; error 63 us; skew 500 ppm
I20260927 22:52:38.756458 10915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:38 dist-test-slave-df53 krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:38.764348 10915 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:52:38.770983 10915 webserver.cc:460] Webserver started at http://127.0.97.2:42255/ using document root <none> and password file <none>
I20260927 22:52:38.771677 10915 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:38.771817 10915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:38.772123 10915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:38.775097 10915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "7a0dd10ab98f4c58bda08cb70e402840"
format_stamp: "Formatted at 2026-09-27 22:52:38 on dist-test-slave-df53"
server_key: "2d4728c21ad6cb84b94dc9855f91e3f2"
server_key_iv: "d4c00474003837c00bdc5f4c60befd11"
server_key_version: "encryptionkey@0"
I20260927 22:52:38.775835 10915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7a0dd10ab98f4c58bda08cb70e402840"
format_stamp: "Formatted at 2026-09-27 22:52:38 on dist-test-slave-df53"
server_key: "2d4728c21ad6cb84b94dc9855f91e3f2"
server_key_iv: "d4c00474003837c00bdc5f4c60befd11"
server_key_version: "encryptionkey@0"
I20260927 22:52:38.780927 10915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:52:38.784509 10930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:38.786041 10915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:52:38.786212 10915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "7a0dd10ab98f4c58bda08cb70e402840"
format_stamp: "Formatted at 2026-09-27 22:52:38 on dist-test-slave-df53"
server_key: "2d4728c21ad6cb84b94dc9855f91e3f2"
server_key_iv: "d4c00474003837c00bdc5f4c60befd11"
server_key_version: "encryptionkey@0"
I20260927 22:52:38.786415 10915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:38.807392 10915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:38.811739 10915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:38.812033 10915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:38.812979 10915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:38.814368 10915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:38.814493 10915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:38.814595 10915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:38.814640 10915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:38.847383 10915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35225
I20260927 22:52:38.847474 11042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35225 every 8 connection(s)
I20260927 22:52:38.848662 10915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:52:38.855326 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 10915
I20260927 22:52:38.855527 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:52:38.855930 388 external_mini_cluster.cc:1428] Setting key 076d02e830fce1ae9367e3af75bbc9d8
Sep 27 22:52:38 dist-test-slave-df53 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:38.869870 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.851416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55529 (local address 127.0.97.62:36015)
0927 22:52:38.851594 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:38.851598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:38.852650 (+ 1052us) server_negotiation.cc:407] Connection header received
0927 22:52:38.854480 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:38.854485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:38.854573 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:38.854684 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:38.856286 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.857037 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.858742 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.859525 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.859668 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.863550 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:38.863559 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:38.863567 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:38.863611 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:38.866498 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:38.867504 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:38.867508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:38.867511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:38.867630 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:38.868110 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:38.868113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:38.868115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:38.868381 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:38.868484 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:38.869389 (+ 905us) server_negotiation.cc:299] Negotiation successful
0927 22:52:38.869561 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:52:38.871335 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:38.851823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36015 (local address 127.0.97.2:55529)
0927 22:52:38.852497 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:38.852532 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:38.854209 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:38.854908 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:38.854916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:38.855393 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:38.856089 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:38.856108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.857201 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.857205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:38.858593 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:38.858605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:38.859757 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:38.859763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:38.859974 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:38.860724 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:38.860754 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:38.863314 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:38.866658 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:38.866670 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:38.866691 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:38.867361 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:38.867740 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:38.867747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:38.867754 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:38.867994 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:38.868514 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:38.868524 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:38.868983 (+ 459us) client_negotiation.cc:769] Sending connection context
0927 22:52:38.869278 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 22:52:38.869892 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":269,"threads_started":1}
I20260927 22:52:38.872792 11043 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36015
I20260927 22:52:38.873143 11043 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:38.873929 11043 heartbeater.cc:511] Master 127.0.97.62:36015 requested a full tablet report, sending...
I20260927 22:52:38.875365 10720 ts_manager.cc:194] Registered new tserver with Master: 7a0dd10ab98f4c58bda08cb70e402840 (127.0.97.2:35225)
I20260927 22:52:38.876259 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55529
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:52:38.909523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36015
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:39.223912 11048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:39.224203 11048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:39.224372 11048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:39.231642 11048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:39.231781 11048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:39.231949 11048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:52:39.241160 11048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:52:39.242331 11048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:39.243538 11048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:39.256106 11053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:39.256065 11054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:39.256138 11056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:39.256979 11048 server_base.cc:1034] running on GCE node
I20260927 22:52:39.257480 11048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:39.258378 11048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:39.259596 11048 hybrid_clock.cc:648] HybridClock initialized: now 1790549559259541 us; error 47 us; skew 500 ppm
I20260927 22:52:39.259883 11048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:39 dist-test-slave-df53 krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:39.262943 11048 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:52:39.264259 11048 webserver.cc:460] Webserver started at http://127.0.97.3:37923/ using document root <none> and password file <none>
I20260927 22:52:39.264897 11048 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:39.265036 11048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:39.265308 11048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:39.268026 11048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "e6bf9935423b4920b54ead3eab7ffc8b"
format_stamp: "Formatted at 2026-09-27 22:52:39 on dist-test-slave-df53"
server_key: "01b8949c2a7f487cf4b1b79ceeaf6fa9"
server_key_iv: "ce8c5a73ea45814bd666e69ac8a6c1e2"
server_key_version: "encryptionkey@0"
I20260927 22:52:39.268754 11048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e6bf9935423b4920b54ead3eab7ffc8b"
format_stamp: "Formatted at 2026-09-27 22:52:39 on dist-test-slave-df53"
server_key: "01b8949c2a7f487cf4b1b79ceeaf6fa9"
server_key_iv: "ce8c5a73ea45814bd666e69ac8a6c1e2"
server_key_version: "encryptionkey@0"
I20260927 22:52:39.273511 11048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:52:39.276741 11063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:39.278177 11048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:39.278362 11048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "e6bf9935423b4920b54ead3eab7ffc8b"
format_stamp: "Formatted at 2026-09-27 22:52:39 on dist-test-slave-df53"
server_key: "01b8949c2a7f487cf4b1b79ceeaf6fa9"
server_key_iv: "ce8c5a73ea45814bd666e69ac8a6c1e2"
server_key_version: "encryptionkey@0"
I20260927 22:52:39.278565 11048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:39.295830 11048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:39.299883 11048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:39.300163 11048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:39.301028 11048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:39.302385 11048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:39.302486 11048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:39.302636 11048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:39.302677 11048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:39.334561 11048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38129
I20260927 22:52:39.334602 11175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38129 every 8 connection(s)
I20260927 22:52:39.335754 11048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:52:39.339803 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 11048
I20260927 22:52:39.340008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:52:39.340427 388 external_mini_cluster.cc:1428] Setting key 2b92beb600556256de9b9db6c4854583
Sep 27 22:52:39 dist-test-slave-df53 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:52:39.356084 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.338403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39425 (local address 127.0.97.62:36015)
0927 22:52:39.338609 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:39.338615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:39.339799 (+ 1184us) server_negotiation.cc:407] Connection header received
0927 22:52:39.342022 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:39.342027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:39.342174 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:39.342290 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:39.344017 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.344735 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.346350 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.346973 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.347105 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.350650 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:39.350655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:39.350658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:39.350693 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:39.353042 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:39.353718 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:39.353722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:39.353724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:39.353890 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:39.354445 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:39.354449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:39.354452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:39.354749 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:39.354908 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:39.355699 (+ 791us) server_negotiation.cc:299] Negotiation successful
0927 22:52:39.355850 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:52:39.356855 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.338944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36015 (local address 127.0.97.3:39425)
0927 22:52:39.339643 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:39.339678 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:39.341771 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:39.342585 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:39.342596 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:39.343314 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:39.343874 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.343891 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.344867 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.344870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.346195 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.346206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.347172 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.347176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.347380 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.348041 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:39.348065 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:39.350475 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:39.353162 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:39.353168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:39.353177 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:39.353603 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:39.354031 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:39.354035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:39.354038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:39.354254 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:39.354899 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:39.354906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:39.355288 (+ 382us) client_negotiation.cc:769] Sending connection context
0927 22:52:39.355597 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 22:52:39.355913 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":278,"threads_started":1}
I20260927 22:52:39.358023 11176 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36015
I20260927 22:52:39.358299 11176 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:39.358940 11176 heartbeater.cc:511] Master 127.0.97.62:36015 requested a full tablet report, sending...
I20260927 22:52:39.360148 10720 ts_manager.cc:194] Registered new tserver with Master: e6bf9935423b4920b54ead3eab7ffc8b (127.0.97.3:38129)
I20260927 22:52:39.360903 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39425
I20260927 22:52:39.367385 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:52:39.382252 10909 heartbeater.cc:503] Master 127.0.97.62:36015 was elected leader, sending a full tablet report...
I20260927 22:52:39.383317 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.371443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.0.97.62:36015)
0927 22:52:39.371594 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:39.371598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:39.372084 (+ 486us) server_negotiation.cc:407] Connection header received
0927 22:52:39.372267 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:39.372272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:39.372340 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:39.372442 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:39.373753 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.374430 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.375629 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.376024 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.376150 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.378316 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:39.378322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:39.378327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:39.378364 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:39.380634 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:39.381238 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:39.381246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:39.381249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:39.381363 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:39.381784 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:39.381787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:39.381789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:39.382135 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:39.382375 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:39.382786 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 22:52:39.383010 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:52:39.387084 10720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:52:39.393841 10720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:52:39.419440 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.405515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.0.97.2:35225)
0927 22:52:39.407144 (+ 1629us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:39.407156 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:39.408376 (+ 1220us) server_negotiation.cc:407] Connection header received
0927 22:52:39.408737 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:39.408743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:39.409012 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:39.409187 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:39.410429 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.412201 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.415782 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.417491 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.417642 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.417895 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:39.418751 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 22:52:39.418976 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1446,"thread_start_us":1346,"threads_started":1}
I20260927 22:52:39.420343 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.406276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38129 (local address 127.0.0.1:41908)
0927 22:52:39.408961 (+ 2685us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:39.408981 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:39.409129 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:39.410252 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:39.410259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:39.410292 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:39.410849 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.410861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.413387 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.413395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.416259 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.416279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.419085 (+ 2806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.419094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.419350 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.419414 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 22:52:39.419525 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:52:39.419704 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2562,"mutex_wait_us":1,"thread_start_us":1655,"threads_started":1}
I20260927 22:52:39.420425 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.405496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35225 (local address 127.0.0.1:48594)
0927 22:52:39.408218 (+ 2722us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:39.408252 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:39.408453 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:39.409658 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:39.409669 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:39.409710 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:39.410183 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.410198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.412388 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.412394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.415580 (+ 3186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.415600 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.418117 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.418126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.418412 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.418481 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 22:52:39.419753 (+ 1272us) client_negotiation.cc:240] Negotiation successful
0927 22:52:39.419930 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2446,"mutex_wait_us":1,"thread_start_us":1684,"threads_started":1}
I20260927 22:52:39.421200 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.406014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.0.97.3:38129)
0927 22:52:39.407694 (+ 1680us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:39.407700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:39.409289 (+ 1589us) server_negotiation.cc:407] Connection header received
0927 22:52:39.409419 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:39.409430 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:39.409728 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:39.409931 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:39.411109 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.413176 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.416512 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.418467 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.418624 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.418905 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:39.420659 (+ 1754us) server_negotiation.cc:299] Negotiation successful
0927 22:52:39.420815 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1530,"thread_start_us":1406,"threads_started":1}
I20260927 22:52:39.424696 11111 tablet_service.cc:1467] Processing CreateTablet for tablet 60850fc79ce64259ae0ba73e42ce08be (DEFAULT_TABLE table=test-table [id=ac3264719dd3428f92108763fec1bc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:39.424696 10978 tablet_service.cc:1467] Processing CreateTablet for tablet 60850fc79ce64259ae0ba73e42ce08be (DEFAULT_TABLE table=test-table [id=ac3264719dd3428f92108763fec1bc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:39.426122 10978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60850fc79ce64259ae0ba73e42ce08be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:39.426122 11111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60850fc79ce64259ae0ba73e42ce08be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:39.428452 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.408784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.0.97.1:35213)
0927 22:52:39.422183 (+ 13399us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:39.422189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:39.422212 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:52:39.422283 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:39.422289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:39.422468 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:39.422662 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:39.423125 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.424075 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.425763 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.427479 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.427634 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.427839 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:39.427995 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 22:52:39.428137 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13243,"thread_start_us":13095,"threads_started":1}
I20260927 22:52:39.428473 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.408616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35213 (local address 127.0.0.1:39314)
0927 22:52:39.420232 (+ 11616us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:39.420259 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:39.420385 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:39.422645 (+ 2260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:39.422649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:39.422673 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:39.422966 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.422974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.424214 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.424220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.425575 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.425632 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.427671 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.427677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.427841 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.427877 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:52:39.427962 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:52:39.428082 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11479,"mutex_wait_us":515,"thread_start_us":11530,"threads_started":1}
I20260927 22:52:39.431861 10844 tablet_service.cc:1467] Processing CreateTablet for tablet 60850fc79ce64259ae0ba73e42ce08be (DEFAULT_TABLE table=test-table [id=ac3264719dd3428f92108763fec1bc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:52:39.433002 10844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60850fc79ce64259ae0ba73e42ce08be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:39.440737 11195 tablet_bootstrap.cc:492] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: Bootstrap starting.
I20260927 22:52:39.441500 11194 tablet_bootstrap.cc:492] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: Bootstrap starting.
I20260927 22:52:39.444720 11196 tablet_bootstrap.cc:492] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde: Bootstrap starting.
I20260927 22:52:39.445560 11195 tablet_bootstrap.cc:654] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:39.446575 11195 log.cc:824] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:39.447402 11196 tablet_bootstrap.cc:654] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:39.448287 11196 log.cc:824] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:39.449554 11194 tablet_bootstrap.cc:654] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:39.450690 11194 log.cc:824] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:39.450846 11195 tablet_bootstrap.cc:492] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: No bootstrap required, opened a new log
I20260927 22:52:39.451134 11195 ts_tablet_manager.cc:1397] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260927 22:52:39.451526 11196 tablet_bootstrap.cc:492] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde: No bootstrap required, opened a new log
I20260927 22:52:39.451784 11196 ts_tablet_manager.cc:1397] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 22:52:39.453526 11194 tablet_bootstrap.cc:492] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: No bootstrap required, opened a new log
I20260927 22:52:39.453822 11194 ts_tablet_manager.cc:1397] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20260927 22:52:39.458508 11195 raft_consensus.cc:348] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0dd10ab98f4c58bda08cb70e402840" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35225 } } peers { permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 } } peers { permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 } }
I20260927 22:52:39.458855 11195 raft_consensus.cc:374] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:39.458993 11195 raft_consensus.cc:729] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6bf9935423b4920b54ead3eab7ffc8b, State: Initialized, Role: FOLLOWER
I20260927 22:52:39.459578 11195 consensus_queue.cc:260] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0dd10ab98f4c58bda08cb70e402840" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35225 } } peers { permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 } } peers { permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 } }
I20260927 22:52:39.460585 11176 heartbeater.cc:503] Master 127.0.97.62:36015 was elected leader, sending a full tablet report...
I20260927 22:52:39.462036 11196 raft_consensus.cc:348] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0dd10ab98f4c58bda08cb70e402840" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35225 } } peers { permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 } } peers { permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 } }
I20260927 22:52:39.462363 11195 ts_tablet_manager.cc:1428] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 22:52:39.462426 11196 raft_consensus.cc:374] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:39.462512 11196 raft_consensus.cc:729] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b173ed5d612e462d9047af2895595bde, State: Initialized, Role: FOLLOWER
I20260927 22:52:39.463065 11196 consensus_queue.cc:260] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0dd10ab98f4c58bda08cb70e402840" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35225 } } peers { permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 } } peers { permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 } }
I20260927 22:52:39.464003 11194 raft_consensus.cc:348] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0dd10ab98f4c58bda08cb70e402840" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35225 } } peers { permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 } } peers { permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 } }
I20260927 22:52:39.464428 11194 raft_consensus.cc:374] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:39.464534 11194 raft_consensus.cc:729] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0dd10ab98f4c58bda08cb70e402840, State: Initialized, Role: FOLLOWER
I20260927 22:52:39.464751 11196 ts_tablet_manager.cc:1428] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20260927 22:52:39.465143 11194 consensus_queue.cc:260] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0dd10ab98f4c58bda08cb70e402840" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35225 } } peers { permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 } } peers { permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 } }
I20260927 22:52:39.467108 11043 heartbeater.cc:503] Master 127.0.97.62:36015 was elected leader, sending a full tablet report...
I20260927 22:52:39.467178 11194 ts_tablet_manager.cc:1428] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
W20260927 22:52:39.587280 11177 tablet.cc:2369] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:52:39.600723 10910 tablet.cc:2369] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:52:39.600723 11044 tablet.cc:2369] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:52:39.666828 11202 raft_consensus.cc:482] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:52:39.667080 11202 raft_consensus.cc:504] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0dd10ab98f4c58bda08cb70e402840" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35225 } } peers { permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 } } peers { permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 } }
I20260927 22:52:39.668781 11202 leader_election.cc:288] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6bf9935423b4920b54ead3eab7ffc8b (127.0.97.3:38129), b173ed5d612e462d9047af2895595bde (127.0.97.1:35213)
I20260927 22:52:39.675462 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.669220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51069 (local address 127.0.97.3:38129)
0927 22:52:39.669473 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:39.669479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:39.670202 (+ 723us) server_negotiation.cc:407] Connection header received
0927 22:52:39.670419 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:39.670423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:39.670480 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:39.670580 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:39.671121 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.671867 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.673787 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.674555 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.674708 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.674813 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:39.675081 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:52:39.675203 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:52:39.675580 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.669319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33963 (local address 127.0.97.1:35213)
0927 22:52:39.669531 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:39.669541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:39.670271 (+ 730us) server_negotiation.cc:407] Connection header received
0927 22:52:39.670471 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:39.670476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:39.670540 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:39.670635 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:52:39.671273 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.671977 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.673873 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.674610 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.674778 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.674899 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:39.675219 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 22:52:39.675338 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:52:39.675688 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.669184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38129 (local address 127.0.97.2:51069)
0927 22:52:39.670080 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:39.670100 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:39.670275 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:39.670608 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:39.670613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:39.670637 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:39.670974 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.670985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.672007 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.672013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.673638 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.673659 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.674762 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.674767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.674922 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.674947 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:52:39.675065 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:52:39.675185 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":35,"thread_start_us":513,"threads_started":1}
I20260927 22:52:39.675730 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.669208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35213 (local address 127.0.97.2:33963)
0927 22:52:39.670154 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:39.670168 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:39.670332 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:39.670660 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:39.670664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:39.670694 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:52:39.671118 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.671126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.672108 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.672116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.673667 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:39.673678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.674856 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.674862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:39.675056 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.675078 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:52:39.675183 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:52:39.675296 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":32,"thread_start_us":669,"threads_started":1}
I20260927 22:52:39.676736 11131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60850fc79ce64259ae0ba73e42ce08be" candidate_uuid: "7a0dd10ab98f4c58bda08cb70e402840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" is_pre_election: true
I20260927 22:52:39.676925 10864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60850fc79ce64259ae0ba73e42ce08be" candidate_uuid: "7a0dd10ab98f4c58bda08cb70e402840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b173ed5d612e462d9047af2895595bde" is_pre_election: true
I20260927 22:52:39.677309 11131 raft_consensus.cc:2454] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0dd10ab98f4c58bda08cb70e402840 in term 0.
I20260927 22:52:39.677428 10864 raft_consensus.cc:2454] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0dd10ab98f4c58bda08cb70e402840 in term 0.
I20260927 22:52:39.678174 10934 leader_election.cc:302] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a0dd10ab98f4c58bda08cb70e402840, b173ed5d612e462d9047af2895595bde, e6bf9935423b4920b54ead3eab7ffc8b; no voters:
I20260927 22:52:39.678634 11202 raft_consensus.cc:2781] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:52:39.678776 11202 raft_consensus.cc:482] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:52:39.678889 11202 raft_consensus.cc:3037] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:39.681063 11202 raft_consensus.cc:504] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0dd10ab98f4c58bda08cb70e402840" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35225 } } peers { permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 } } peers { permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 } }
I20260927 22:52:39.681668 11202 leader_election.cc:288] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [CANDIDATE]: Term 1 election: Requested vote from peers e6bf9935423b4920b54ead3eab7ffc8b (127.0.97.3:38129), b173ed5d612e462d9047af2895595bde (127.0.97.1:35213)
I20260927 22:52:39.681979 11131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60850fc79ce64259ae0ba73e42ce08be" candidate_uuid: "7a0dd10ab98f4c58bda08cb70e402840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6bf9935423b4920b54ead3eab7ffc8b"
I20260927 22:52:39.682127 10864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60850fc79ce64259ae0ba73e42ce08be" candidate_uuid: "7a0dd10ab98f4c58bda08cb70e402840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b173ed5d612e462d9047af2895595bde"
I20260927 22:52:39.682180 11131 raft_consensus.cc:3037] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:39.682332 10864 raft_consensus.cc:3037] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:39.685508 11131 raft_consensus.cc:2454] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0dd10ab98f4c58bda08cb70e402840 in term 1.
I20260927 22:52:39.685505 10864 raft_consensus.cc:2454] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0dd10ab98f4c58bda08cb70e402840 in term 1.
I20260927 22:52:39.685880 10934 leader_election.cc:302] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0dd10ab98f4c58bda08cb70e402840, e6bf9935423b4920b54ead3eab7ffc8b; no voters:
I20260927 22:52:39.686187 11202 raft_consensus.cc:2781] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:39.686636 11202 raft_consensus.cc:686] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 1 LEADER]: Becoming Leader. State: Replica: 7a0dd10ab98f4c58bda08cb70e402840, State: Running, Role: LEADER
I20260927 22:52:39.687100 11202 consensus_queue.cc:237] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0dd10ab98f4c58bda08cb70e402840" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35225 } } peers { permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 } } peers { permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 } }
I20260927 22:52:39.691172 10720 catalog_manager.cc:5495] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a0dd10ab98f4c58bda08cb70e402840 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7a0dd10ab98f4c58bda08cb70e402840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0dd10ab98f4c58bda08cb70e402840" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 } health_report { overall_health: UNKNOWN } } }
I20260927 22:52:39.706745 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.701653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.0.97.2:35225)
0927 22:52:39.701847 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:39.701855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:39.701968 (+ 113us) server_negotiation.cc:407] Connection header received
0927 22:52:39.702130 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:39.702134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:39.702216 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:39.702334 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:52:39.702648 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.703440 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.704694 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.705084 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.705179 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.705656 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:52:39.706136 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:52:39.706243 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:39.706378 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 22:52:39.706452 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:52:39.714787 10864 raft_consensus.cc:1261] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde [term 1 FOLLOWER]: Refusing update from remote peer 7a0dd10ab98f4c58bda08cb70e402840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:39.714787 11131 raft_consensus.cc:1261] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b [term 1 FOLLOWER]: Refusing update from remote peer 7a0dd10ab98f4c58bda08cb70e402840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:52:39.715852 11205 consensus_queue.cc:1035] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6bf9935423b4920b54ead3eab7ffc8b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:39.716223 11202 consensus_queue.cc:1035] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b173ed5d612e462d9047af2895595bde" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:52:39.730701 11210 mvcc.cc:205] Tried to move back new op lower bound from 7334090996581150720 to 7334090996482232320. Current Snapshot: MvccSnapshot[applied={T|T < 7334090996581150720}]
I20260927 22:52:39.740897 10720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:39.741158 10720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:52:39.744436 10720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 555e5af87fbc4586afe97bec4d933831: Sending DeleteTablet for 3 replicas of tablet 60850fc79ce64259ae0ba73e42ce08be
I20260927 22:52:39.745427 10978 tablet_service.cc:1514] Processing DeleteTablet for tablet 60850fc79ce64259ae0ba73e42ce08be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48594
I20260927 22:52:39.745553 11111 tablet_service.cc:1514] Processing DeleteTablet for tablet 60850fc79ce64259ae0ba73e42ce08be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41908
I20260927 22:52:39.746522 10844 tablet_service.cc:1514] Processing DeleteTablet for tablet 60850fc79ce64259ae0ba73e42ce08be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:52:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39314
I20260927 22:52:39.746866 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 10781
I20260927 22:52:39.756724 11218 tablet_replica.cc:330] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: stopping tablet replica
I20260927 22:52:39.757304 11217 tablet_replica.cc:330] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: stopping tablet replica
I20260927 22:52:39.757922 11218 raft_consensus.cc:2229] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:39.758325 11217 raft_consensus.cc:2229] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:52:39.758857 11218 raft_consensus.cc:2258] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:39.759495 11217 raft_consensus.cc:2258] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:39.761801 11218 ts_tablet_manager.cc:1905] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:52:39.767526 11217 ts_tablet_manager.cc:1905] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:52:39 dist-test-slave-df53 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:52:39.774799 11217 ts_tablet_manager.cc:1918] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:39.774959 11217 log.cc:1196] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/60850fc79ce64259ae0ba73e42ce08be
I20260927 22:52:39.775492 11217 ts_tablet_manager.cc:1939] T 60850fc79ce64259ae0ba73e42ce08be P 7a0dd10ab98f4c58bda08cb70e402840: Deleting consensus metadata
I20260927 22:52:39.776094 11219 tablet_replica.cc:330] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde: stopping tablet replica
I20260927 22:52:39.776678 11219 raft_consensus.cc:2229] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:52:39.777257 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.748972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.0.97.1:35213)
0927 22:52:39.749147 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:39.749153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:39.758326 (+ 9173us) server_negotiation.cc:407] Connection header received
0927 22:52:39.758806 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:39.758816 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:39.758964 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:39.759063 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:39.761147 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.762327 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.764162 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.764786 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.764883 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.769910 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:39.769926 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:39.769944 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:39.769982 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:39.773640 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:39.774474 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:39.774483 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:39.774487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:39.774630 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:39.775078 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:39.775085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:39.775090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:39.775629 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:39.775945 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:39.776631 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 22:52:39.776873 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:52:39.777288 10704 catalog_manager.cc:4841] TS 7a0dd10ab98f4c58bda08cb70e402840 (127.0.97.2:35225): tablet 60850fc79ce64259ae0ba73e42ce08be (table test-table [id=ac3264719dd3428f92108763fec1bc15]) successfully deleted
I20260927 22:52:39.778493 10904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:39.777236 11219 raft_consensus.cc:2258] T 60850fc79ce64259ae0ba73e42ce08be P b173ed5d612e462d9047af2895595bde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:52:39.780803 11218 ts_tablet_manager.cc:1918] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:52:39.780972 11218 log.cc:1196] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/60850fc79ce64259ae0ba73e42ce08be
I20260927 22:52:39.781515 11218 ts_tablet_manager.cc:1939] T 60850fc79ce64259ae0ba73e42ce08be P e6bf9935423b4920b54ead3eab7ffc8b: Deleting consensus metadata
I20260927 22:52:39.783434 10707 catalog_manager.cc:4841] TS e6bf9935423b4920b54ead3eab7ffc8b (127.0.97.3:38129): tablet 60850fc79ce64259ae0ba73e42ce08be (table test-table [id=ac3264719dd3428f92108763fec1bc15]) successfully deleted
I20260927 22:52:39.891451 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 10781
I20260927 22:52:39.915210 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 10915
W20260927 22:52:39.915246 10704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35213 (error 108)
I20260927 22:52:39.917953 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.917530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35213 (local address 127.0.0.1:39324)
0927 22:52:39.917724 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:39.917815 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 22:52:39.918279 10704 catalog_manager.cc:4584] TS b173ed5d612e462d9047af2895595bde (127.0.97.1:35213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60850fc79ce64259ae0ba73e42ce08be: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35213: connect: Connection refused (error 111)
Sep 27 22:52:39 dist-test-slave-df53 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:52:39.932053 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.916076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.0.97.2:35225)
0927 22:52:39.916274 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:39.916280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:39.916526 (+ 246us) server_negotiation.cc:407] Connection header received
0927 22:52:39.916733 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:39.916739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:39.916819 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:39.916932 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:39.918717 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.919534 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.921322 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:39.921845 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:39.921957 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:39.926156 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:39.926169 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:39.926182 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:39.926216 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:39.929185 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:39.929923 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:39.929935 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:39.929940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:39.930076 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:39.930516 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:39.930521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:39.930525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:39.930940 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:39.931299 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:39.931500 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 22:52:39.931724 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:52:39.932802 11038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:39.972057 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.971593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35213 (local address 127.0.0.1:39340)
0927 22:52:39.971811 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:39.971918 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 22:52:39.973171 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:39.972806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35213 (local address 127.0.0.1:39356)
0927 22:52:39.972982 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:39.973059 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 22:52:40.033844 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:40.033287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35213 (local address 127.0.0.1:39364)
0927 22:52:40.033500 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:40.033651 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 22:52:40.034808 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:40.034487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35213 (local address 127.0.0.1:39376)
0927 22:52:40.034633 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:40.034705 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:52:40.045545 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 10915
I20260927 22:52:40.064064 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 11048
Sep 27 22:52:40 dist-test-slave-df53 krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:52:40.078590 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:40.064812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.0.97.3:38129)
0927 22:52:40.065028 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:40.065034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:40.065054 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:52:40.065249 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:40.065257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:40.065325 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:40.065448 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:40.066729 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:40.067464 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:40.068700 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:40.069084 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:40.069222 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:40.073349 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:40.073362 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:40.073378 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:40.073413 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:40.075927 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:40.076596 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:40.076603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:40.076606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:40.076736 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:40.077106 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:40.077112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:40.077115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:40.077447 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:40.077758 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:40.078062 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 22:52:40.078267 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:52:40.079270 11171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:40.114992 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:40.114537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35213 (local address 127.0.0.1:39386)
0927 22:52:40.114742 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:40.114827 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 22:52:40.116014 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:40.115653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35213 (local address 127.0.0.1:39388)
0927 22:52:40.115835 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:40.115911 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 22:52:40.167038 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 11048
I20260927 22:52:40.184907 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 10688
I20260927 22:52:40.186247 10750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:40.292447 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:40.291927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35213 (local address 127.0.0.1:39394)
0927 22:52:40.292132 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:40.292252 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:52:40.292855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 10688
2026-09-27T22:52:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4238 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:40 dist-test-slave-df53 krb5kdc[11228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:40 dist-test-slave-df53 krb5kdc[11228](info): set up 2 sockets
Sep 27 22:52:40 dist-test-slave-df53 krb5kdc[11228](info): commencing operation
krb5kdc: starting...
W20260927 22:52:41.394476 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:52:41 dist-test-slave-df53 krb5kdc[11228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 22:52:41.700748 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIqwSmZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/postgres -l logfile start
2026-09-27 22:52:44.130 UTC [11258] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:52:44.130 UTC [11258] LOG: listening on IPv4 address "127.0.97.20", port 41183
2026-09-27 22:52:44.133 UTC [11258] LOG: listening on Unix socket "/tmp/.s.PGSQL.41183"
2026-09-27 22:52:44.155 UTC [11261] LOG: database system was shut down at 2026-09-27 22:52:42 UTC
2026-09-27 22:52:44.158 UTC [11258] LOG: database system is ready to accept connections
I20260927 22:52:45.126075 388 mini_postgres.cc:96] Postgres bound to 41183
2026-09-27 22:52:45.142 UTC [11269] FATAL: database "slave" does not exist
127.0.97.20:41183 - accepting connections
I20260927 22:52:45.143462 388 mini_ranger.cc:161] Starting Ranger...
I20260927 22:52:45.167268 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:52:45.630164 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:52:45.630296 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-admin
2026-09-27 22:52:45,904 [I] DB FLAVOR :POSTGRES
2026-09-27 22:52:45,904 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:52:45,905 [I] Checking connection..
2026-09-27 22:52:45,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:52:46,153 [I] Checking connection passed.
2026-09-27 22:52:46,154 [I] --------- Verifying version history table ---------
2026-09-27 22:52:46,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:52:46,411 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:52:46,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:52:46,654 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:52:46,655 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:52:46,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:52:46.908 UTC [11333] WARNING: there is no transaction in progress
2026-09-27 22:52:46,914 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:52:46,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:52:47,173 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:52:47,173 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:52:47,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:52:47,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:52:47,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-df53.c.gcp-upstream.internal','N') ;"
2026-09-27 22:52:47,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:52:48,160 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:52:48,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:52:48,386 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:52:48,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:52:48,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:52:48,896 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:52:48,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:52:49.823 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:49.839 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:49.854 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:49.868 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:49.883 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:49.900 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:49.954 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:49.960 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:49.973 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:49.983 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:49.993 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:50.007 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:50.016 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:50.022 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:50.028 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:50.035 UTC [11441] WARNING: there is no transaction in progress
2026-09-27 22:52:50,657 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:52:50,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:52:50,907 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:52:50,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:52:51,138 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:52:51,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:52:51,389 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:52:51,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:52:51,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:52:51,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-df53.c.gcp-upstream.internal';"
2026-09-27 22:52:52,118 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:52:52,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:52:52,371 [I] DB_PATCHES have already been applied
I20260927 22:52:52.377931 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIqwSmZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:52:52.378016 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 22:52:52.391923 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:57239
Sep 27, 2026 10:52:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:52:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:52:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:52:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:52:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:52:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:52:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:52:53.522341 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.379s user 0.003s sys 0.035s
Sep 27, 2026 10:53:12 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:53:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 10:53:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:53:13 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:53:13 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:53:14.451958 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:53:14 dist-test-slave-df53 krb5kdc[11228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549594, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:53:14.632504 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:53:14.653841 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:53:15.526084 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:53:15.526274 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-kms
I20260927 22:53:15.526407 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:41183
2026-09-27 22:53:15,931 [I] DB FLAVOR :POSTGRES
2026-09-27 22:53:15,932 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:53:15,932 [I] Checking connection
2026-09-27 22:53:15,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:53:16,160 [I] connection success
2026-09-27 22:53:16,161 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:53:16,161 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:53:16,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:53:16,383 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:53:16,384 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:53:16,384 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:53:16,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:41183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:53:16,633 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:53:17.066092 388 mini_ranger_kms.cc:310] Created kms service
I20260927 22:53:17.285406 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:53:17.330106 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:53:17 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549594, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:53:17.732399 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:53:17.732743 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:53:17.732806 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 22:53:17.753746 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11621
I20260927 22:53:17.753931 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:39955
Sep 27, 2026 10:53:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:53:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:53:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:53:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:53:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:53:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 22:53:19 dist-test-slave-df53 krb5kdc[11228](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:20.102325 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.470s user 0.113s sys 0.313s
I20260927 22:53:20.102588 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:53:20.102797 388 mini_ranger_kms.cc:401] 127.0.97.20:39955/kms/v1/keys
Sep 27 22:53:20 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:53:20 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:53:22 dist-test-slave-df53 krb5kdc[11228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:53:22.873139 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32831
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32831
--ranger_config_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39955/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:23.192199 11672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:23.192513 11672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:23.192662 11672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:23.199673 11672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:23.199816 11672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:23.199910 11672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:23.200011 11672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:23.200093 11672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:23.209795 11672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:39955/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32831
--ranger_config_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:53:23.210968 11672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:23.212198 11672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:23.223691 11680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:23.223785 11678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:23.223752 11677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:23.224335 11672 server_base.cc:1034] running on GCE node
I20260927 22:53:23.225028 11672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:23.226254 11672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:23.227473 11672 hybrid_clock.cc:648] HybridClock initialized: now 1790549603227437 us; error 44 us; skew 500 ppm
I20260927 22:53:23.227801 11672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:23 dist-test-slave-df53 krb5kdc[11228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:23.231037 11672 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:53:23.232406 11672 webserver.cc:460] Webserver started at http://127.0.97.62:33055/ using document root <none> and password file <none>
I20260927 22:53:23.233039 11672 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:23.233173 11672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:23.233445 11672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:53:23 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:53:23.287855 11672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "10b26fb622a94fdbb6eba0a017bd5db3"
format_stamp: "Formatted at 2026-09-27 22:53:23 on dist-test-slave-df53"
server_key: "00dceae57d3ff0f2d19fd6227cb08537"
server_key_iv: "f21d2e6946ea1370c55f8f48e4f8db97"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:23.288650 11672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "10b26fb622a94fdbb6eba0a017bd5db3"
format_stamp: "Formatted at 2026-09-27 22:53:23 on dist-test-slave-df53"
server_key: "00dceae57d3ff0f2d19fd6227cb08537"
server_key_iv: "f21d2e6946ea1370c55f8f48e4f8db97"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:23.293807 11672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:53:23.309090 11687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:23.310685 11672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:53:23.310957 11672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "10b26fb622a94fdbb6eba0a017bd5db3"
format_stamp: "Formatted at 2026-09-27 22:53:23 on dist-test-slave-df53"
server_key: "00dceae57d3ff0f2d19fd6227cb08537"
server_key_iv: "f21d2e6946ea1370c55f8f48e4f8db97"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:23.311179 11672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:23.340003 11672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:23.343770 11672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:23.344061 11672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:23.363037 11672 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32831
I20260927 22:53:23.363083 11738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32831 every 8 connection(s)
I20260927 22:53:23.364303 11672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:23.365401 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 11672
I20260927 22:53:23.365630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:53:23.366220 388 external_mini_cluster.cc:1428] Setting key 2af6c0cf5715dad8fbb5fc08569aaf1d
I20260927 22:53:23.368671 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:53:23 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:53:23.382858 11739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3: Bootstrap starting.
I20260927 22:53:23.387128 11739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:23.388476 11739 log.cc:824] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:23.390784 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:23.368049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.0.97.62:32831)
0927 22:53:23.368728 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:23.368739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:23.368801 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:53:23.370408 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:23.370428 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:23.370770 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:23.371091 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:23.372713 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.374492 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.376428 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.377065 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.377192 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:23.381806 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:23.381818 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:23.381837 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:23.381881 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:23.386078 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.386721 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.386730 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.386738 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.386949 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.387311 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.387318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.387321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.388032 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:23.388334 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:23.388664 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 22:53:23.389123 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":308,"threads_started":1}
I20260927 22:53:23.392295 11739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3: No bootstrap required, opened a new log
I20260927 22:53:23.400146 11739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b26fb622a94fdbb6eba0a017bd5db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32831 } }
I20260927 22:53:23.400490 11739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:23.400616 11739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b26fb622a94fdbb6eba0a017bd5db3, State: Initialized, Role: FOLLOWER
I20260927 22:53:23.401232 11739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b26fb622a94fdbb6eba0a017bd5db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32831 } }
I20260927 22:53:23.401463 11739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:23.401647 11739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:23.401829 11739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:23.404263 11739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b26fb622a94fdbb6eba0a017bd5db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32831 } }
I20260927 22:53:23.404752 11739 leader_election.cc:302] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10b26fb622a94fdbb6eba0a017bd5db3; no voters:
I20260927 22:53:23.405328 11739 leader_election.cc:288] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:23.405419 11744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:23.406690 11744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [term 1 LEADER]: Becoming Leader. State: Replica: 10b26fb622a94fdbb6eba0a017bd5db3, State: Running, Role: LEADER
I20260927 22:53:23.407267 11744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b26fb622a94fdbb6eba0a017bd5db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32831 } }
I20260927 22:53:23.407446 11739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:23.410360 11745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10b26fb622a94fdbb6eba0a017bd5db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b26fb622a94fdbb6eba0a017bd5db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32831 } } }
I20260927 22:53:23.410789 11745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:23.410668 11746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10b26fb622a94fdbb6eba0a017bd5db3. Latest consensus state: current_term: 1 leader_uuid: "10b26fb622a94fdbb6eba0a017bd5db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b26fb622a94fdbb6eba0a017bd5db3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32831 } } }
I20260927 22:53:23.410920 11746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:23.412802 11753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:23.418334 11753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:23.419104 11739 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:53:23.435575 11753 catalog_manager.cc:1347] Generated new cluster ID: 517a30da5fad41cb8f575bc89c311b87
I20260927 22:53:23.435745 11753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:23.456825 11753 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:23.458590 11753 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:23.469136 11753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3: Generated new TSK 0
I20260927 22:53:23.470094 11753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:53:24.253810 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32831
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:53:24 dist-test-slave-df53 krb5kdc[11228](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:53:24.670356 11784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:24.670642 11784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:24.670812 11784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:24.678190 11784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:24.678319 11784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:24.678512 11784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:53:24.689160 11784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:39955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:53:24.690779 11784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:24.692473 11784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:24.711208 11794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:24.712188 11795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:24.718868 11797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:24.719606 11784 server_base.cc:1034] running on GCE node
I20260927 22:53:24.720512 11784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:24.725579 11784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:24.726912 11784 hybrid_clock.cc:648] HybridClock initialized: now 1790549604726804 us; error 94 us; skew 500 ppm
I20260927 22:53:24.727281 11784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:24 dist-test-slave-df53 krb5kdc[11228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:24.731459 11784 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:53:24.733355 11784 webserver.cc:460] Webserver started at http://127.0.97.1:45653/ using document root <none> and password file <none>
I20260927 22:53:24.734236 11784 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:24.734373 11784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:24.734740 11784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:53:24 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:53:24.755162 11784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "be257e8b6952456a917a94c0f3fbb56e"
format_stamp: "Formatted at 2026-09-27 22:53:24 on dist-test-slave-df53"
server_key: "28bb8826d3f09570120c03b6e28aa921"
server_key_iv: "7b097f2cff85c0924db04c0594581172"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:24.756214 11784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "be257e8b6952456a917a94c0f3fbb56e"
format_stamp: "Formatted at 2026-09-27 22:53:24 on dist-test-slave-df53"
server_key: "28bb8826d3f09570120c03b6e28aa921"
server_key_iv: "7b097f2cff85c0924db04c0594581172"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:24.762800 11784 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 22:53:24.789960 11804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:24.791985 11784 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.005s
I20260927 22:53:24.792176 11784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "be257e8b6952456a917a94c0f3fbb56e"
format_stamp: "Formatted at 2026-09-27 22:53:24 on dist-test-slave-df53"
server_key: "28bb8826d3f09570120c03b6e28aa921"
server_key_iv: "7b097f2cff85c0924db04c0594581172"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:24.792554 11784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:24.820446 11784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:24.824780 11784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:24.825086 11784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:24.826074 11784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:24.827380 11784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:24.827453 11784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:24.827610 11784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:24.827667 11784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:24.920336 11784 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35033
I20260927 22:53:24.920792 11916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35033 every 8 connection(s)
I20260927 22:53:24.922233 11784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:24.925022 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 11784
I20260927 22:53:24.925266 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:24.926278 388 external_mini_cluster.cc:1428] Setting key 0291a20cf9dabf5a3826299cc8a0830b
Sep 27 22:53:24 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:53:24.978039 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:24.927583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45133 (local address 127.0.97.62:32831)
0927 22:53:24.946167 (+ 18584us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:24.946173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:24.948685 (+ 2512us) server_negotiation.cc:407] Connection header received
0927 22:53:24.952173 (+ 3488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:24.952180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:24.952278 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:24.952359 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:24.957570 (+ 5211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:24.959162 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:24.961930 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:24.962565 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:24.962663 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:24.970129 (+ 7466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:24.970137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:24.970141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:24.970189 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:24.974118 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:24.975129 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:24.975136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:24.975139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:24.975300 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:24.975871 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:24.975877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:24.975882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:24.976272 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:24.976403 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:24.977389 (+ 986us) server_negotiation.cc:299] Negotiation successful
0927 22:53:24.977644 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18442,"thread_start_us":22014,"threads_started":1}
I20260927 22:53:24.980306 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:24.927937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32831 (local address 127.0.97.1:45133)
0927 22:53:24.947601 (+ 19664us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:24.947660 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:24.951802 (+ 4142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:24.953699 (+ 1897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:24.953713 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:24.955679 (+ 1966us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:24.956525 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:24.956553 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:24.959373 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:24.959378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:24.961755 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:24.961770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:24.963012 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:24.963019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:24.963329 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:24.964146 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:24.964185 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:24.969802 (+ 5617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:24.974310 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:24.974320 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:24.974339 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:24.974980 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:24.975454 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:24.975459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:24.975463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:24.975735 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:24.976496 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:24.976505 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:24.976946 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 22:53:24.978263 (+ 1317us) client_negotiation.cc:240] Negotiation successful
0927 22:53:24.978822 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19318,"thread_start_us":20673,"threads_started":1}
I20260927 22:53:24.982640 11917 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32831
I20260927 22:53:24.982971 11917 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:24.983779 11917 heartbeater.cc:511] Master 127.0.97.62:32831 requested a full tablet report, sending...
I20260927 22:53:24.986267 11704 ts_manager.cc:194] Registered new tserver with Master: be257e8b6952456a917a94c0f3fbb56e (127.0.97.1:35033)
I20260927 22:53:24.988162 11704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:53:25.011116 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32831
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:53:25 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:53:25 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 22:53:25.438835 11923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:25.439276 11923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:25.439482 11923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:25.450680 11923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:25.450891 11923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:25.451128 11923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:53:25.466776 11923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:39955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:53:25.468667 11923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:25.470438 11923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:25.504073 11930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:25.509997 11931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:25.512847 11934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:25.513293 11923 server_base.cc:1034] running on GCE node
I20260927 22:53:25.514287 11923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:25.518800 11923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:25.520094 11923 hybrid_clock.cc:648] HybridClock initialized: now 1790549605519990 us; error 98 us; skew 500 ppm
I20260927 22:53:25.520469 11923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:25 dist-test-slave-df53 krb5kdc[11228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:25.524694 11923 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:53:25.582084 11923 webserver.cc:460] Webserver started at http://127.0.97.2:34275/ using document root <none> and password file <none>
I20260927 22:53:25.583284 11923 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:25.583520 11923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:25.583982 11923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:53:25 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:53:25.609236 11923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "91a30d398cdb4ee78f5a32ddf709631a"
format_stamp: "Formatted at 2026-09-27 22:53:25 on dist-test-slave-df53"
server_key: "9faf194f69580da9079d7cb17fdd6ead"
server_key_iv: "67995279aeb390847469c1087f251bc7"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:25.610323 11923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "91a30d398cdb4ee78f5a32ddf709631a"
format_stamp: "Formatted at 2026-09-27 22:53:25 on dist-test-slave-df53"
server_key: "9faf194f69580da9079d7cb17fdd6ead"
server_key_iv: "67995279aeb390847469c1087f251bc7"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:25.627300 11923 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.004s sys 0.007s
I20260927 22:53:25.686096 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:25.689330 11923 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.004s sys 0.001s
I20260927 22:53:25.689661 11923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "91a30d398cdb4ee78f5a32ddf709631a"
format_stamp: "Formatted at 2026-09-27 22:53:25 on dist-test-slave-df53"
server_key: "9faf194f69580da9079d7cb17fdd6ead"
server_key_iv: "67995279aeb390847469c1087f251bc7"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:25.690006 11923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:25.777711 11923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:25.782950 11923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:25.783332 11923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:25.790668 11923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:25.799124 11923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:25.799279 11923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:25.799441 11923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:25.799542 11923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:25.845069 11923 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35231
I20260927 22:53:25.845111 12059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35231 every 8 connection(s)
I20260927 22:53:25.846664 11923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:25.857059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 11923
I20260927 22:53:25.857215 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:25.857744 388 external_mini_cluster.cc:1428] Setting key b5853365437227832db7569b55f74487
Sep 27 22:53:25 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:53:25.890501 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:25.849886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32831 (local address 127.0.97.2:41049)
0927 22:53:25.866615 (+ 16729us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:25.866667 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:25.869019 (+ 2352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:25.870107 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:25.870120 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:25.870719 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:25.871498 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:25.871522 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:25.872989 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:25.872995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:25.874940 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:25.874955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:25.876537 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:25.876543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:25.876816 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:25.877780 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:25.877810 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:25.880544 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:25.884518 (+ 3974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:25.884540 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:25.884559 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:25.886334 (+ 1775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:25.886797 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:25.886802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:25.886805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:25.887074 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:25.887815 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:25.887837 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:25.888266 (+ 429us) client_negotiation.cc:769] Sending connection context
0927 22:53:25.888558 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 22:53:25.889074 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16422,"thread_start_us":16189,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:53:25.892221 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:25.850507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41049 (local address 127.0.97.62:32831)
0927 22:53:25.867480 (+ 16973us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:25.867486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:25.867509 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:53:25.869474 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:25.869480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:25.869559 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:25.869678 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:25.871699 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:25.872793 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:25.875123 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:25.876150 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:25.876265 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:25.880799 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:25.880806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:25.880811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:25.880857 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:25.884338 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:25.886485 (+ 2147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:25.886493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:25.886497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:25.886664 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:25.887204 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:25.887209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:25.887212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:25.887587 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:25.887726 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:25.891498 (+ 3772us) server_negotiation.cc:299] Negotiation successful
0927 22:53:25.891827 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16846,"thread_start_us":16990,"threads_started":1}
I20260927 22:53:25.894099 12060 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32831
I20260927 22:53:25.894388 12060 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:25.895185 12060 heartbeater.cc:511] Master 127.0.97.62:32831 requested a full tablet report, sending...
I20260927 22:53:25.896955 11704 ts_manager.cc:194] Registered new tserver with Master: 91a30d398cdb4ee78f5a32ddf709631a (127.0.97.2:35231)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 22:53:25.898079 11704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:53:25.933665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:39955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32831
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:53:25.992219 11917 heartbeater.cc:503] Master 127.0.97.62:32831 was elected leader, sending a full tablet report...
W20260927 22:53:26.296600 12066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:26.296977 12066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:26.297178 12066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:26.305330 12066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:26.305452 12066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:26.305676 12066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:53:26.315397 12066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:39955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:53:26.316653 12066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:26.317945 12066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:26.330915 12071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:26.330937 12074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:26.330865 12072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:26.331534 12066 server_base.cc:1034] running on GCE node
I20260927 22:53:26.332147 12066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:26.333410 12066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:26.334641 12066 hybrid_clock.cc:648] HybridClock initialized: now 1790549606334635 us; error 91 us; skew 500 ppm
I20260927 22:53:26.334959 12066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:26 dist-test-slave-df53 krb5kdc[11228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:26.337888 12066 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:53:26.339242 12066 webserver.cc:460] Webserver started at http://127.0.97.3:45841/ using document root <none> and password file <none>
I20260927 22:53:26.339874 12066 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:26.340019 12066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:26.340308 12066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:53:26 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:53:26.355585 12066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "6df62c7018b7452699c3b52b3ea0f379"
format_stamp: "Formatted at 2026-09-27 22:53:26 on dist-test-slave-df53"
server_key: "637fdadf296214a7641ddf48e377fd9b"
server_key_iv: "5b05a22765bd0ee62b85d7e2042e706f"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:26.356397 12066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6df62c7018b7452699c3b52b3ea0f379"
format_stamp: "Formatted at 2026-09-27 22:53:26 on dist-test-slave-df53"
server_key: "637fdadf296214a7641ddf48e377fd9b"
server_key_iv: "5b05a22765bd0ee62b85d7e2042e706f"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:26.361334 12066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:53:26.376649 12081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:26.378155 12066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:53:26.378332 12066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "6df62c7018b7452699c3b52b3ea0f379"
format_stamp: "Formatted at 2026-09-27 22:53:26 on dist-test-slave-df53"
server_key: "637fdadf296214a7641ddf48e377fd9b"
server_key_iv: "5b05a22765bd0ee62b85d7e2042e706f"
server_key_version: "kuduclusterkey@0"
I20260927 22:53:26.378538 12066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:26.395712 12066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:26.399859 12066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:26.400138 12066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:26.401005 12066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:26.402746 12066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:26.402818 12066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:26.402944 12066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:26.402992 12066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:26.435679 12066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35281
I20260927 22:53:26.435715 12193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35281 every 8 connection(s)
I20260927 22:53:26.436896 12066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:26.442615 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 12066
I20260927 22:53:26.442816 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:26.443250 388 external_mini_cluster.cc:1428] Setting key 4955f0f503483e8d4e37f562c95dd7b1
Sep 27 22:53:26 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:53:26.456925 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.440115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54471 (local address 127.0.97.62:32831)
0927 22:53:26.440577 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.440583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.440606 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:53:26.442310 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.442316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.442406 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.442536 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.444113 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.445560 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.447940 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.448358 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.448451 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.451596 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.451601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.451604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.451632 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.454212 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.454868 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.454876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.454880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.454995 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.455375 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.455382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.455386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.455631 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.455782 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.456367 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.456572 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":863,"threads_started":1}
I20260927 22:53:26.457871 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.439755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32831 (local address 127.0.97.3:54471)
0927 22:53:26.440409 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:26.440452 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:26.442064 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:26.442811 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:26.442819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:26.443252 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:26.443950 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.443981 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.445847 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.445854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.447809 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.447820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.448547 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.448553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.448735 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.449348 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:26.449369 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:26.451407 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:26.454338 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:26.454346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:26.454361 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:26.454754 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:26.455095 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:26.455103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:26.455107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:26.455270 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:26.455754 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:26.455762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:26.456047 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 22:53:26.456296 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 22:53:26.456738 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":438,"threads_started":1}
I20260927 22:53:26.459221 12194 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32831
I20260927 22:53:26.459503 12194 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:26.460140 12194 heartbeater.cc:511] Master 127.0.97.62:32831 requested a full tablet report, sending...
I20260927 22:53:26.461354 11704 ts_manager.cc:194] Registered new tserver with Master: 6df62c7018b7452699c3b52b3ea0f379 (127.0.97.3:35281)
I20260927 22:53:26.462162 11704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54471
I20260927 22:53:26.471240 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:53:26.488371 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.475616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.0.97.62:32831)
0927 22:53:26.475789 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.475796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.476606 (+ 810us) server_negotiation.cc:407] Connection header received
0927 22:53:26.476804 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.476816 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.476882 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.476983 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.478367 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.479053 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.480378 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.480815 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.480951 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.483532 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.483538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.483541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.483568 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.485919 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.486544 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.486551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.486555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.486668 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.487063 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.487068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.487072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.487319 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.487574 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.487876 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.488078 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:53:26.492152 11704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:26.499742 11704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:26.524806 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.511661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35231 (local address 127.0.0.1:34290)
0927 22:53:26.515672 (+ 4011us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:26.515709 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:26.515966 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:26.516956 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:26.516963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:26.517011 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:26.518430 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.518446 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.520128 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.520133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.522131 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.522151 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.523842 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.523849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.524030 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.524075 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:53:26.524224 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:53:26.524412 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3713,"mutex_wait_us":1072,"spinlock_wait_cycles":2688,"thread_start_us":3583,"threads_started":1}
I20260927 22:53:26.525758 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.512312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.0.97.1:35033)
0927 22:53:26.515088 (+ 2776us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.515098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.516133 (+ 1035us) server_negotiation.cc:407] Connection header received
0927 22:53:26.516387 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.516393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.516566 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.516711 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:26.518211 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.519757 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.522187 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.523655 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.523784 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.524003 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.524514 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.524685 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":2287,"threads_started":1}
I20260927 22:53:26.526119 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.512027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.0.97.2:35231)
0927 22:53:26.512690 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.512699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.515806 (+ 3107us) server_negotiation.cc:407] Connection header received
0927 22:53:26.516252 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.516264 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.516521 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.516711 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:26.518601 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.519984 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.522275 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.523655 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.523784 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.523974 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.524303 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.524480 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260927 22:53:26.526135 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.516201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35281 (local address 127.0.0.1:51474)
0927 22:53:26.518114 (+ 1913us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:26.518131 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:26.518274 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:26.519123 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:26.519128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:26.519150 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:26.519483 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.519495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.521358 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.521365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.523338 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.523352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.525396 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.525401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.525506 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.525526 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 22:53:26.525699 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 22:53:26.525814 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1799,"thread_start_us":1657,"threads_started":1}
I20260927 22:53:26.526216 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.516481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.0.97.3:35281)
0927 22:53:26.517987 (+ 1506us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.517995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.518229 (+ 234us) server_negotiation.cc:407] Connection header received
0927 22:53:26.518446 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.518451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.518698 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.518874 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:26.519681 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.521214 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.523507 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.524852 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.524983 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.525282 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.525738 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.525877 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":719,"threads_started":1}
I20260927 22:53:26.526630 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.512070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35033 (local address 127.0.0.1:57962)
0927 22:53:26.516030 (+ 3960us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:26.516056 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:26.516224 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:26.516956 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:26.516963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:26.517013 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:26.517972 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.517985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.519932 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.519939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.522026 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.522043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.524148 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.524153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.524325 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.524361 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:53:26.526131 (+ 1770us) client_negotiation.cc:240] Negotiation successful
0927 22:53:26.526288 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3841,"mutex_wait_us":93,"spinlock_wait_cycles":7552,"thread_start_us":3673,"threads_started":1}
I20260927 22:53:26.529654 11995 tablet_service.cc:1467] Processing CreateTablet for tablet 1be7426aeeb64016bdc7fddfcdb6fb35 (DEFAULT_TABLE table=test-table [id=8d57e89dcb354ffe81f3e5b356dab03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:26.529874 11852 tablet_service.cc:1467] Processing CreateTablet for tablet 1be7426aeeb64016bdc7fddfcdb6fb35 (DEFAULT_TABLE table=test-table [id=8d57e89dcb354ffe81f3e5b356dab03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:26.529874 12129 tablet_service.cc:1467] Processing CreateTablet for tablet 1be7426aeeb64016bdc7fddfcdb6fb35 (DEFAULT_TABLE table=test-table [id=8d57e89dcb354ffe81f3e5b356dab03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:26.530849 11995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be7426aeeb64016bdc7fddfcdb6fb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:26.531168 12129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be7426aeeb64016bdc7fddfcdb6fb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:26.531168 11852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be7426aeeb64016bdc7fddfcdb6fb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:26.541941 12213 tablet_bootstrap.cc:492] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: Bootstrap starting.
I20260927 22:53:26.545202 12214 tablet_bootstrap.cc:492] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: Bootstrap starting.
I20260927 22:53:26.545562 12213 tablet_bootstrap.cc:654] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:26.546015 12215 tablet_bootstrap.cc:492] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: Bootstrap starting.
I20260927 22:53:26.546839 12213 log.cc:824] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:26.549386 12213 tablet_bootstrap.cc:492] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: No bootstrap required, opened a new log
I20260927 22:53:26.549746 12213 ts_tablet_manager.cc:1397] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:53:26.549811 12215 tablet_bootstrap.cc:654] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:26.551064 12215 log.cc:824] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:26.551589 12214 tablet_bootstrap.cc:654] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:26.552977 12214 log.cc:824] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:26.555738 12215 tablet_bootstrap.cc:492] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: No bootstrap required, opened a new log
I20260927 22:53:26.556051 12215 ts_tablet_manager.cc:1397] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260927 22:53:26.556535 12214 tablet_bootstrap.cc:492] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: No bootstrap required, opened a new log
I20260927 22:53:26.556773 12214 ts_tablet_manager.cc:1397] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260927 22:53:26.558790 12213 raft_consensus.cc:348] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a30d398cdb4ee78f5a32ddf709631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35231 } } peers { permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 } } peers { permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 } }
I20260927 22:53:26.559180 12213 raft_consensus.cc:374] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:26.559309 12213 raft_consensus.cc:729] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a30d398cdb4ee78f5a32ddf709631a, State: Initialized, Role: FOLLOWER
I20260927 22:53:26.559881 12213 consensus_queue.cc:260] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a30d398cdb4ee78f5a32ddf709631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35231 } } peers { permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 } } peers { permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 } }
I20260927 22:53:26.561307 12060 heartbeater.cc:503] Master 127.0.97.62:32831 was elected leader, sending a full tablet report...
I20260927 22:53:26.561896 12213 ts_tablet_manager.cc:1428] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 22:53:26.565269 12214 raft_consensus.cc:348] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a30d398cdb4ee78f5a32ddf709631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35231 } } peers { permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 } } peers { permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 } }
I20260927 22:53:26.565307 12215 raft_consensus.cc:348] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a30d398cdb4ee78f5a32ddf709631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35231 } } peers { permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 } } peers { permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 } }
I20260927 22:53:26.565716 12214 raft_consensus.cc:374] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:26.565755 12215 raft_consensus.cc:374] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:26.565836 12214 raft_consensus.cc:729] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be257e8b6952456a917a94c0f3fbb56e, State: Initialized, Role: FOLLOWER
I20260927 22:53:26.565851 12215 raft_consensus.cc:729] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df62c7018b7452699c3b52b3ea0f379, State: Initialized, Role: FOLLOWER
I20260927 22:53:26.566393 12215 consensus_queue.cc:260] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a30d398cdb4ee78f5a32ddf709631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35231 } } peers { permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 } } peers { permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 } }
I20260927 22:53:26.566473 12214 consensus_queue.cc:260] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a30d398cdb4ee78f5a32ddf709631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35231 } } peers { permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 } } peers { permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 } }
I20260927 22:53:26.567471 12194 heartbeater.cc:503] Master 127.0.97.62:32831 was elected leader, sending a full tablet report...
I20260927 22:53:26.568876 12214 ts_tablet_manager.cc:1428] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260927 22:53:26.569205 12215 ts_tablet_manager.cc:1428] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260927 22:53:26.588253 12219 raft_consensus.cc:482] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:26.588588 12219 raft_consensus.cc:504] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a30d398cdb4ee78f5a32ddf709631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35231 } } peers { permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 } } peers { permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 } }
I20260927 22:53:26.590291 12219 leader_election.cc:288] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be257e8b6952456a917a94c0f3fbb56e (127.0.97.1:35033), 6df62c7018b7452699c3b52b3ea0f379 (127.0.97.3:35281)
I20260927 22:53:26.597471 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.590683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51437 (local address 127.0.97.1:35033)
0927 22:53:26.590872 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.590880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.591467 (+ 587us) server_negotiation.cc:407] Connection header received
0927 22:53:26.591799 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.591804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.591880 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.591941 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:26.593034 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.593929 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.595720 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.596512 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.596607 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.596703 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.597083 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.597225 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:53:26.597543 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.590588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35033 (local address 127.0.97.2:51437)
0927 22:53:26.591376 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:26.591395 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:26.591581 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:26.592092 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:26.592097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:26.592124 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:26.592883 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.592898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.594047 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.594053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.595568 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.595579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.596718 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.596724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.596927 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.596953 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:53:26.597026 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 22:53:26.597129 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":1,"thread_start_us":457,"threads_started":1}
I20260927 22:53:26.597683 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.591668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36949 (local address 127.0.97.3:35281)
0927 22:53:26.591864 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.591872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.592100 (+ 228us) server_negotiation.cc:407] Connection header received
0927 22:53:26.592302 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.592309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.592368 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.592457 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:26.593305 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.594151 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.595848 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.596617 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.596742 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.596835 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.597208 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.597338 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:53:26.597831 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.591455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35281 (local address 127.0.97.2:36949)
0927 22:53:26.592005 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:26.592018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:26.592167 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:26.592470 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:26.592474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:26.592496 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:26.593162 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.593176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.594297 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.594301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.595707 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.595721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.596819 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.596823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.597047 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.597079 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:53:26.597169 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 22:53:26.597281 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":55,"thread_start_us":269,"threads_started":1}
I20260927 22:53:26.598528 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be7426aeeb64016bdc7fddfcdb6fb35" candidate_uuid: "91a30d398cdb4ee78f5a32ddf709631a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be257e8b6952456a917a94c0f3fbb56e" is_pre_election: true
I20260927 22:53:26.598769 12149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be7426aeeb64016bdc7fddfcdb6fb35" candidate_uuid: "91a30d398cdb4ee78f5a32ddf709631a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6df62c7018b7452699c3b52b3ea0f379" is_pre_election: true
I20260927 22:53:26.599174 11872 raft_consensus.cc:2454] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a30d398cdb4ee78f5a32ddf709631a in term 0.
I20260927 22:53:26.599289 12149 raft_consensus.cc:2454] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a30d398cdb4ee78f5a32ddf709631a in term 0.
I20260927 22:53:26.599843 11951 leader_election.cc:302] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91a30d398cdb4ee78f5a32ddf709631a, be257e8b6952456a917a94c0f3fbb56e; no voters:
I20260927 22:53:26.600224 12219 raft_consensus.cc:2781] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:26.600373 12219 raft_consensus.cc:482] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:26.600486 12219 raft_consensus.cc:3037] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:26.602679 12219 raft_consensus.cc:504] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a30d398cdb4ee78f5a32ddf709631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35231 } } peers { permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 } } peers { permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 } }
I20260927 22:53:26.603696 12219 leader_election.cc:288] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [CANDIDATE]: Term 1 election: Requested vote from peers be257e8b6952456a917a94c0f3fbb56e (127.0.97.1:35033), 6df62c7018b7452699c3b52b3ea0f379 (127.0.97.3:35281)
I20260927 22:53:26.603936 11872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be7426aeeb64016bdc7fddfcdb6fb35" candidate_uuid: "91a30d398cdb4ee78f5a32ddf709631a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be257e8b6952456a917a94c0f3fbb56e"
I20260927 22:53:26.604164 11872 raft_consensus.cc:3037] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:26.604185 12149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be7426aeeb64016bdc7fddfcdb6fb35" candidate_uuid: "91a30d398cdb4ee78f5a32ddf709631a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6df62c7018b7452699c3b52b3ea0f379"
I20260927 22:53:26.604353 12149 raft_consensus.cc:3037] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379 [term 0 FOLLOWER]: Advancing to term 1
W20260927 22:53:26.604904 12061 tablet.cc:2369] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:26.606568 12149 raft_consensus.cc:2454] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a30d398cdb4ee78f5a32ddf709631a in term 1.
I20260927 22:53:26.606689 11872 raft_consensus.cc:2454] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a30d398cdb4ee78f5a32ddf709631a in term 1.
I20260927 22:53:26.606956 11951 leader_election.cc:302] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6df62c7018b7452699c3b52b3ea0f379, 91a30d398cdb4ee78f5a32ddf709631a; no voters:
I20260927 22:53:26.607203 12219 raft_consensus.cc:2781] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:26.607744 12219 raft_consensus.cc:686] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 1 LEADER]: Becoming Leader. State: Replica: 91a30d398cdb4ee78f5a32ddf709631a, State: Running, Role: LEADER
I20260927 22:53:26.608228 12219 consensus_queue.cc:237] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a30d398cdb4ee78f5a32ddf709631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35231 } } peers { permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 } } peers { permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 } }
I20260927 22:53:26.612671 11703 catalog_manager.cc:5495] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a reported cstate change: term changed from 0 to 1, leader changed from <none> to 91a30d398cdb4ee78f5a32ddf709631a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "91a30d398cdb4ee78f5a32ddf709631a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91a30d398cdb4ee78f5a32ddf709631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 } health_report { overall_health: UNKNOWN } } }
I20260927 22:53:26.643249 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.638125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.0.97.2:35231)
0927 22:53:26.638297 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.638301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.638399 (+ 98us) server_negotiation.cc:407] Connection header received
0927 22:53:26.638549 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.638552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.638645 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.638762 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:26.639069 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.639784 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.641473 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.641886 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.641975 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.642362 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:26.642829 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:26.642929 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.643010 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.643066 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:53:26.650470 12149 raft_consensus.cc:1261] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379 [term 1 FOLLOWER]: Refusing update from remote peer 91a30d398cdb4ee78f5a32ddf709631a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:26.650470 11872 raft_consensus.cc:1261] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e [term 1 FOLLOWER]: Refusing update from remote peer 91a30d398cdb4ee78f5a32ddf709631a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:26.651360 12224 consensus_queue.cc:1035] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6df62c7018b7452699c3b52b3ea0f379" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:26.651733 12219 consensus_queue.cc:1035] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [LEADER]: Connected to new peer: Peer: permanent_uuid: "be257e8b6952456a917a94c0f3fbb56e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:26.662111 12227 mvcc.cc:205] Tried to move back new op lower bound from 7334091188831354880 to 7334091188671254528. Current Snapshot: MvccSnapshot[applied={T|T < 7334091188831354880}]
I20260927 22:53:26.663228 12228 mvcc.cc:205] Tried to move back new op lower bound from 7334091188831354880 to 7334091188671254528. Current Snapshot: MvccSnapshot[applied={T|T < 7334091188831354880}]
I20260927 22:53:26.676304 11704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:26.676556 11704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:26.679828 11704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10b26fb622a94fdbb6eba0a017bd5db3: Sending DeleteTablet for 3 replicas of tablet 1be7426aeeb64016bdc7fddfcdb6fb35
I20260927 22:53:26.680855 11995 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be7426aeeb64016bdc7fddfcdb6fb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34290
I20260927 22:53:26.681166 12129 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be7426aeeb64016bdc7fddfcdb6fb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51474
I20260927 22:53:26.681174 11852 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be7426aeeb64016bdc7fddfcdb6fb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57962
I20260927 22:53:26.683610 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 11784
I20260927 22:53:26.696255 12236 tablet_replica.cc:330] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: stopping tablet replica
I20260927 22:53:26.696699 12236 raft_consensus.cc:2229] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:26.697422 12236 raft_consensus.cc:2258] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:26.698167 12237 tablet_replica.cc:330] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: stopping tablet replica
I20260927 22:53:26.698637 12237 raft_consensus.cc:2229] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:26.699110 12237 raft_consensus.cc:2258] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:26.699586 12238 tablet_replica.cc:330] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: stopping tablet replica
I20260927 22:53:26.699981 12238 raft_consensus.cc:2229] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:26.700443 12238 raft_consensus.cc:2258] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:26.700572 12236 ts_tablet_manager.cc:1905] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:26.701184 12237 ts_tablet_manager.cc:1905] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:26.702406 12238 ts_tablet_manager.cc:1905] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:26.708029 12237 ts_tablet_manager.cc:1918] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:26.708215 12237 log.cc:1196] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/1be7426aeeb64016bdc7fddfcdb6fb35
I20260927 22:53:26.708758 12237 ts_tablet_manager.cc:1939] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 6df62c7018b7452699c3b52b3ea0f379: Deleting consensus metadata
I20260927 22:53:26.709833 12238 ts_tablet_manager.cc:1918] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:26.710024 12238 log.cc:1196] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/1be7426aeeb64016bdc7fddfcdb6fb35
I20260927 22:53:26.710546 12238 ts_tablet_manager.cc:1939] T 1be7426aeeb64016bdc7fddfcdb6fb35 P be257e8b6952456a917a94c0f3fbb56e: Deleting consensus metadata
I20260927 22:53:26.710769 11691 catalog_manager.cc:4841] TS 6df62c7018b7452699c3b52b3ea0f379 (127.0.97.3:35281): tablet 1be7426aeeb64016bdc7fddfcdb6fb35 (table test-table [id=8d57e89dcb354ffe81f3e5b356dab03c]) successfully deleted
Sep 27 22:53:26 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:53:26.712309 11691 catalog_manager.cc:4841] TS be257e8b6952456a917a94c0f3fbb56e (127.0.97.1:35033): tablet 1be7426aeeb64016bdc7fddfcdb6fb35 (table test-table [id=8d57e89dcb354ffe81f3e5b356dab03c]) successfully deleted
I20260927 22:53:26.715164 12236 ts_tablet_manager.cc:1918] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:26.715328 12236 log.cc:1196] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/1be7426aeeb64016bdc7fddfcdb6fb35
I20260927 22:53:26.715842 12236 ts_tablet_manager.cc:1939] T 1be7426aeeb64016bdc7fddfcdb6fb35 P 91a30d398cdb4ee78f5a32ddf709631a: Deleting consensus metadata
I20260927 22:53:26.717381 11689 catalog_manager.cc:4841] TS 91a30d398cdb4ee78f5a32ddf709631a (127.0.97.2:35231): tablet 1be7426aeeb64016bdc7fddfcdb6fb35 (table test-table [id=8d57e89dcb354ffe81f3e5b356dab03c]) successfully deleted
I20260927 22:53:26.718721 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.684432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.0.97.1:35033)
0927 22:53:26.684596 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.684602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.690445 (+ 5843us) server_negotiation.cc:407] Connection header received
0927 22:53:26.691559 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.691569 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.691708 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.691801 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.695780 (+ 3979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.697250 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.702631 (+ 5381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.703286 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.703382 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.712791 (+ 9409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.712802 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.712814 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.712851 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.715888 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.716603 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.716611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.716615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.716748 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.717104 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.717109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.717112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.717535 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.717866 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.718173 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.718379 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:53:26.719377 11912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:26.812311 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 11784
I20260927 22:53:26.829537 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 11923
Sep 27 22:53:26 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:53:26.845973 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.830390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.0.97.2:35231)
0927 22:53:26.830596 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.830611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.830633 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:53:26.830757 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.830762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.830848 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.830957 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.832121 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.833342 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.834741 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.835394 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.835520 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.839686 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.839697 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.839713 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.839743 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.842953 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.843766 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.843773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.843775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.843907 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.844297 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.844301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.844304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.844749 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.845055 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.845397 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.845630 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:53:26.846771 12055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:26.955170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 11923
I20260927 22:53:26.972127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 12066
Sep 27 22:53:26 dist-test-slave-df53 krb5kdc[11228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:53:26.986733 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.972861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.0.97.3:35281)
0927 22:53:26.973017 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.973021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.973041 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:53:26.973247 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.973252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.973324 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.973430 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.974725 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.975444 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.976696 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.977057 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.977146 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.981231 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.981244 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.981259 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.981289 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.983978 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.984658 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.984666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.984668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.984804 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.985182 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.985188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.985192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.985559 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.985941 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.986197 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.986407 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:53:26.987313 12189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:27.074854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 12066
I20260927 22:53:27.090610 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 11672
I20260927 22:53:27.091701 11734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:27.201895 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 11672
I20260927 22:53:27.224433 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:53:27.559002 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:53:27.559238 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:53:27.903813 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:53:27.904 UTC [11258] LOG: received smart shutdown request
2026-09-27 22:53:27.907 UTC [11258] LOG: background worker "logical replication launcher" (PID 11267) exited with exit code 1
2026-09-27 22:53:27.907 UTC [11262] LOG: shutting down
2026-09-27 22:53:28.066 UTC [11258] LOG: database system is shut down
2026-09-27T22:53:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47898 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:28 dist-test-slave-df53 krb5kdc[12252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:28 dist-test-slave-df53 krb5kdc[12252](info): set up 2 sockets
Sep 27 22:53:28 dist-test-slave-df53 krb5kdc[12252](info): commencing operation
krb5kdc: starting...
W20260927 22:53:29.285903 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:53:29 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 22:53:29.580530 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIqwSmZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/postgres -l logfile start
2026-09-27 22:53:31.904 UTC [12282] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:53:31.904 UTC [12282] LOG: listening on IPv4 address "127.0.97.20", port 48715
2026-09-27 22:53:31.908 UTC [12282] LOG: listening on Unix socket "/tmp/.s.PGSQL.48715"
2026-09-27 22:53:31.928 UTC [12285] LOG: database system was shut down at 2026-09-27 22:53:30 UTC
2026-09-27 22:53:31.931 UTC [12282] LOG: database system is ready to accept connections
I20260927 22:53:32.901535 388 mini_postgres.cc:96] Postgres bound to 48715
2026-09-27 22:53:32.918 UTC [12293] FATAL: database "slave" does not exist
127.0.97.20:48715 - accepting connections
I20260927 22:53:32.919039 388 mini_ranger.cc:161] Starting Ranger...
I20260927 22:53:32.948320 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:53:33.410349 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:53:33.410472 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-admin
2026-09-27 22:53:33,716 [I] DB FLAVOR :POSTGRES
2026-09-27 22:53:33,716 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:53:33,717 [I] Checking connection..
2026-09-27 22:53:33,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:53:33,966 [I] Checking connection passed.
2026-09-27 22:53:33,966 [I] --------- Verifying version history table ---------
2026-09-27 22:53:33,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:53:34,212 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:53:34,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:53:34,443 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:53:34,443 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:53:34,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:53:34.694 UTC [12357] WARNING: there is no transaction in progress
2026-09-27 22:53:34,700 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:53:34,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:53:34,928 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:53:34,928 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:53:34,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:53:35,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:53:35,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-df53.c.gcp-upstream.internal','N') ;"
2026-09-27 22:53:35,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:53:35,905 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:53:35,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:53:36,155 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:53:36,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:53:36,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:53:36,600 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:53:36,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:53:37.502 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.518 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.533 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.548 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.565 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.584 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.636 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.642 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.655 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.665 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.674 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.687 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.696 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.702 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.708 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:37.714 UTC [12465] WARNING: there is no transaction in progress
2026-09-27 22:53:38,328 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:53:38,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:53:38,572 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:53:38,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:53:38,802 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:53:38,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:53:39,055 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:53:39,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:53:39,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:53:39,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-df53.c.gcp-upstream.internal';"
2026-09-27 22:53:39,776 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:53:39,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:53:40,032 [I] DB_PATCHES have already been applied
I20260927 22:53:40.038864 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIqwSmZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:53:40.038971 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 22:53:40.052845 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:56869
Sep 27, 2026 10:53:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:53:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:53:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:53:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:53:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:53:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:53:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:53:41.180413 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.261s user 0.000s sys 0.041s
Sep 27, 2026 10:53:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:53:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 10:53:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:54:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:54:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:54:01.480850 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:54:01 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549641, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:54:01.634478 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:54:01.653659 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:54:02.471709 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:54:02.471832 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-kms
I20260927 22:54:02.471884 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48715
2026-09-27 22:54:02,820 [I] DB FLAVOR :POSTGRES
2026-09-27 22:54:02,820 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:54:02,821 [I] Checking connection
2026-09-27 22:54:02,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:54:03,055 [I] connection success
2026-09-27 22:54:03,056 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:54:03,056 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:54:03,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:54:03,280 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:54:03,280 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:54:03,280 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:54:03,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:54:03,531 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:54:03.978382 388 mini_ranger_kms.cc:310] Created kms service
I20260927 22:54:04.200794 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:54:04.245062 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:54:04 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549641, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:04.648176 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:54:04.648485 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIqwSmZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIqwSmZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:54:04.648553 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 22:54:04.663269 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12645
I20260927 22:54:04.663404 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:58355
Sep 27, 2026 10:54:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:54:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:54:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:54:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:54:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:54:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 22:54:06 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:06.934110 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.300s user 0.079s sys 0.281s
I20260927 22:54:06.934376 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:54:06.934562 388 mini_ranger_kms.cc:401] 127.0.97.20:58355/kms/v1/keys
Sep 27 22:54:07 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:54:07 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 22:54:09 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:54:09.697985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44613
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44613
--ranger_config_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58355/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:10.016593 12696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:10.016855 12696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:10.017004 12696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:10.024165 12696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:10.024268 12696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:10.024364 12696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:10.024462 12696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:10.024549 12696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:10.034523 12696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:58355/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44613
--ranger_config_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:10.035673 12696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:10.036953 12696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:10.048256 12702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:10.048317 12701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:10.048274 12704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:10.049170 12696 server_base.cc:1034] running on GCE node
I20260927 22:54:10.049930 12696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:10.051218 12696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:10.052440 12696 hybrid_clock.cc:648] HybridClock initialized: now 1790549650052404 us; error 50 us; skew 500 ppm
I20260927 22:54:10.052798 12696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:10 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:10.063880 12696 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:54:10.065246 12696 webserver.cc:460] Webserver started at http://127.0.97.62:39651/ using document root <none> and password file <none>
I20260927 22:54:10.065863 12696 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:10.065995 12696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:10.066279 12696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:54:10 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:10.121809 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "c16633baddc746c598a2bca105ab3832"
format_stamp: "Formatted at 2026-09-27 22:54:10 on dist-test-slave-df53"
server_key: "0aa352b484968c83b634e97a28ad1b14"
server_key_iv: "000bc98c0e484b332823fbeedfb9933d"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:10.122828 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "c16633baddc746c598a2bca105ab3832"
format_stamp: "Formatted at 2026-09-27 22:54:10 on dist-test-slave-df53"
server_key: "0aa352b484968c83b634e97a28ad1b14"
server_key_iv: "000bc98c0e484b332823fbeedfb9933d"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:10.128599 12696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:54:10.146759 12711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:10.148437 12696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:54:10.148622 12696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "c16633baddc746c598a2bca105ab3832"
format_stamp: "Formatted at 2026-09-27 22:54:10 on dist-test-slave-df53"
server_key: "0aa352b484968c83b634e97a28ad1b14"
server_key_iv: "000bc98c0e484b332823fbeedfb9933d"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:10.148866 12696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:10.183600 12696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:10.187173 12696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:10.187418 12696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:10.204154 12696 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44613
I20260927 22:54:10.204180 12762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44613 every 8 connection(s)
I20260927 22:54:10.205407 12696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:10.209007 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:10.211431 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 12696
I20260927 22:54:10.211643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:54:10.212314 388 external_mini_cluster.cc:1428] Setting key 2089789eaebca6a99c1ec3500287313e
I20260927 22:54:10.220307 12763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: Bootstrap starting.
I20260927 22:54:10.223266 12763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:10.224342 12763 log.cc:824] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:54:10 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:10.227067 12763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: No bootstrap required, opened a new log
I20260927 22:54:10.233934 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:10.214278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.0.97.62:44613)
0927 22:54:10.215035 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:10.215046 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:10.215085 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:54:10.216997 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:10.217021 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:10.217455 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:10.218034 (+ 579us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:10.219372 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.220709 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.222099 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.222795 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.222986 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:10.226835 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:10.226843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:10.226857 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:10.226895 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:10.230116 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:10.230773 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:10.230780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:10.230785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:10.230931 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:10.231317 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:10.231320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:10.231322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:10.231796 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:10.232116 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:10.232549 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 22:54:10.232922 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":342,"threads_started":1}
I20260927 22:54:10.235221 12763 raft_consensus.cc:348] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } }
I20260927 22:54:10.235666 12763 raft_consensus.cc:374] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:10.235764 12763 raft_consensus.cc:729] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16633baddc746c598a2bca105ab3832, State: Initialized, Role: FOLLOWER
I20260927 22:54:10.236402 12763 consensus_queue.cc:260] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } }
I20260927 22:54:10.236706 12763 raft_consensus.cc:388] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:10.236853 12763 raft_consensus.cc:482] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:10.237016 12763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:10.239107 12763 raft_consensus.cc:504] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } }
I20260927 22:54:10.239517 12763 leader_election.cc:302] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c16633baddc746c598a2bca105ab3832; no voters:
I20260927 22:54:10.240072 12763 leader_election.cc:288] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:54:10.240154 12768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:10.240818 12768 raft_consensus.cc:686] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 1 LEADER]: Becoming Leader. State: Replica: c16633baddc746c598a2bca105ab3832, State: Running, Role: LEADER
I20260927 22:54:10.241302 12768 consensus_queue.cc:237] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } }
I20260927 22:54:10.242219 12763 sys_catalog.cc:564] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:10.244163 12770 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c16633baddc746c598a2bca105ab3832. Latest consensus state: current_term: 1 leader_uuid: "c16633baddc746c598a2bca105ab3832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } } }
I20260927 22:54:10.244138 12769 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c16633baddc746c598a2bca105ab3832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } } }
I20260927 22:54:10.244581 12769 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:10.244581 12770 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:10.247125 12774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:10.251478 12774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:10.257129 12763 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:54:10.266973 12774 catalog_manager.cc:1347] Generated new cluster ID: c877cba986784e049f86a5c7f37817e1
I20260927 22:54:10.267398 12774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:10.299070 12774 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:54:10.301074 12774 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:10.309379 12774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: Generated new TSK 0
I20260927 22:54:10.310480 12774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:54:11.051321 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44613
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:54:11 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:54:11.450592 12808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:11.451076 12808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:11.451277 12808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:11.462577 12808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:11.462697 12808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:11.462908 12808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:54:11.478050 12808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:58355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:11.479734 12808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:11.481436 12808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:11.497380 12818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:11.498996 12819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:11.502578 12821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:11.503189 12808 server_base.cc:1034] running on GCE node
I20260927 22:54:11.503785 12808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:11.504652 12808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:11.505916 12808 hybrid_clock.cc:648] HybridClock initialized: now 1790549651505870 us; error 36 us; skew 500 ppm
I20260927 22:54:11.506265 12808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:11 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:11.509903 12808 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:54:11.521003 12808 webserver.cc:460] Webserver started at http://127.0.97.1:44575/ using document root <none> and password file <none>
I20260927 22:54:11.521790 12808 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:11.521924 12808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:11.522289 12808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:54:11 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:11.541131 12808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "6d1867f251fc4a8cbc0967d205238415"
format_stamp: "Formatted at 2026-09-27 22:54:11 on dist-test-slave-df53"
server_key: "a29be560fb32646a6f21c60ec7ed5e22"
server_key_iv: "2838bbca1d0f0bc42c9f9eb1642abe88"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:11.542138 12808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6d1867f251fc4a8cbc0967d205238415"
format_stamp: "Formatted at 2026-09-27 22:54:11 on dist-test-slave-df53"
server_key: "a29be560fb32646a6f21c60ec7ed5e22"
server_key_iv: "2838bbca1d0f0bc42c9f9eb1642abe88"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:11.548861 12808 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:54:11.582271 12828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:11.584218 12808 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.016s sys 0.001s
I20260927 22:54:11.584465 12808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "6d1867f251fc4a8cbc0967d205238415"
format_stamp: "Formatted at 2026-09-27 22:54:11 on dist-test-slave-df53"
server_key: "a29be560fb32646a6f21c60ec7ed5e22"
server_key_iv: "2838bbca1d0f0bc42c9f9eb1642abe88"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:11.584837 12808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:11.610812 12808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:11.615886 12808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:11.616266 12808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:11.625064 12808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:11.626910 12808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:11.627002 12808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:11.627167 12808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:11.627233 12808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:54:11 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:11.910974 12808 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39809
I20260927 22:54:11.910995 12940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39809 every 8 connection(s)
I20260927 22:54:11.912778 12808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:11.919438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 12808
I20260927 22:54:11.919672 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:54:11.920648 388 external_mini_cluster.cc:1428] Setting key 88b1cf4ad1184e40450bec24edc77408
Sep 27 22:54:11 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:54:11.977738 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:11.915605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34077 (local address 127.0.97.62:44613)
0927 22:54:11.952316 (+ 36711us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:11.952326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:11.954662 (+ 2336us) server_negotiation.cc:407] Connection header received
0927 22:54:11.958008 (+ 3346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:11.958014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:11.958108 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:11.958215 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:11.960329 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.962930 (+ 2601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.965234 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.965890 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.965982 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:11.970519 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:11.970525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:11.970530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:11.970578 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:11.974047 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:11.974932 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:11.974938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:11.974941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:11.975086 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:11.975633 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:11.975638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:11.975641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:11.975978 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:11.976097 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:11.977095 (+ 998us) server_negotiation.cc:299] Negotiation successful
0927 22:54:11.977325 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36518,"thread_start_us":36710,"threads_started":1}
I20260927 22:54:11.979929 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:11.916358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44613 (local address 127.0.97.1:34077)
0927 22:54:11.954170 (+ 37812us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:11.954229 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:11.957507 (+ 3278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:11.958657 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:11.958670 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:11.959302 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:11.960122 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:11.960148 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.963127 (+ 2979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.963131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:11.965050 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:11.965069 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.966305 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.966311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:11.966571 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:11.967327 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:11.967356 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:11.970272 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:11.974197 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:11.974207 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:11.974226 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:11.974800 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:11.975232 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:11.975237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:11.975241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:11.975502 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:11.976207 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:11.976215 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:11.976625 (+ 410us) client_negotiation.cc:769] Sending connection context
0927 22:54:11.977956 (+ 1331us) client_negotiation.cc:240] Negotiation successful
0927 22:54:11.978513 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37498,"thread_start_us":37236,"threads_started":1}
I20260927 22:54:11.981966 12941 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44613
I20260927 22:54:11.982306 12941 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:11.983119 12941 heartbeater.cc:511] Master 127.0.97.62:44613 requested a full tablet report, sending...
I20260927 22:54:11.985527 12728 ts_manager.cc:194] Registered new tserver with Master: 6d1867f251fc4a8cbc0967d205238415 (127.0.97.1:39809)
I20260927 22:54:11.987468 12728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:54:12.005177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44613
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:54:12 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 22:54:12.430279 12947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:12.430565 12947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:12.430737 12947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:12.437795 12947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:12.437923 12947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:12.438078 12947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:54:12.447450 12947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:58355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:12.448606 12947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:12.449878 12947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:12.462852 12962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:12.462898 12964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:12.462942 12961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:12.467144 12947 server_base.cc:1034] running on GCE node
I20260927 22:54:12.467694 12947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:12.468573 12947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:12.469810 12947 hybrid_clock.cc:648] HybridClock initialized: now 1790549652469747 us; error 50 us; skew 500 ppm
I20260927 22:54:12.470118 12947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:12 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:12.473002 12947 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:54:12.474402 12947 webserver.cc:460] Webserver started at http://127.0.97.2:39449/ using document root <none> and password file <none>
I20260927 22:54:12.475036 12947 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:12.475170 12947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:12.475453 12947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:54:12 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:12.489012 12947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "b9b1045dfc5745139a733f463ba4c020"
format_stamp: "Formatted at 2026-09-27 22:54:12 on dist-test-slave-df53"
server_key: "2ef2e3030dd37531c5c3091c7f9aeeec"
server_key_iv: "02eb01dc06e7db36f43753735a0ba825"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:12.489926 12947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b9b1045dfc5745139a733f463ba4c020"
format_stamp: "Formatted at 2026-09-27 22:54:12 on dist-test-slave-df53"
server_key: "2ef2e3030dd37531c5c3091c7f9aeeec"
server_key_iv: "02eb01dc06e7db36f43753735a0ba825"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:12.495242 12947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:54:12.510134 12971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:12.511646 12947 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260927 22:54:12.511825 12947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "b9b1045dfc5745139a733f463ba4c020"
format_stamp: "Formatted at 2026-09-27 22:54:12 on dist-test-slave-df53"
server_key: "2ef2e3030dd37531c5c3091c7f9aeeec"
server_key_iv: "02eb01dc06e7db36f43753735a0ba825"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:12.512058 12947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:12.542683 12947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:12.546722 12947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:12.547026 12947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:12.548023 12947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:12.549474 12947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:12.549573 12947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:12.549762 12947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:12.549870 12947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:12.585197 12947 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38923
I20260927 22:54:12.585304 13083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38923 every 8 connection(s)
I20260927 22:54:12.586521 12947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:12.595595 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 12947
I20260927 22:54:12.595790 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:54:12.596318 388 external_mini_cluster.cc:1428] Setting key 04d8c92927f95f1befe9233655b0c4c6
Sep 27 22:54:12 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:12.612280 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:12.589180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57069 (local address 127.0.97.62:44613)
0927 22:54:12.589722 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:12.589729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:12.590775 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 22:54:12.592692 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:12.592698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:12.592795 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:12.592925 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:12.594357 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.595546 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.597764 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.598343 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.600137 (+ 1794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:12.604925 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:12.604931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:12.604937 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:12.604976 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:12.608442 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:12.609444 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:12.609449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:12.609454 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:12.609634 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:12.610237 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:12.610251 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:12.610255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:12.610589 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:12.610748 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:12.611649 (+ 901us) server_negotiation.cc:299] Negotiation successful
0927 22:54:12.611873 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":251,"threads_started":1}
I20260927 22:54:12.613771 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:12.589730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44613 (local address 127.0.97.2:57069)
0927 22:54:12.590589 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:12.590633 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:12.592414 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:12.593113 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:12.593121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:12.593545 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:12.594173 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:12.594189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.595731 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.595737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:12.597572 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:12.597584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.600516 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.600525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:12.600982 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:12.601831 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:12.601860 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:12.604663 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:12.608590 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:12.608602 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:12.608621 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:12.609300 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:12.609850 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:12.609855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:12.609859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:12.610084 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:12.610743 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:12.610754 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:12.611192 (+ 438us) client_negotiation.cc:769] Sending connection context
0927 22:54:12.611500 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 22:54:12.612039 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":363,"threads_started":1}
I20260927 22:54:12.615391 13084 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44613
I20260927 22:54:12.615715 13084 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:12.616641 13084 heartbeater.cc:511] Master 127.0.97.62:44613 requested a full tablet report, sending...
I20260927 22:54:12.618396 12728 ts_manager.cc:194] Registered new tserver with Master: b9b1045dfc5745139a733f463ba4c020 (127.0.97.2:38923)
I20260927 22:54:12.619524 12728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57069
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:54:12.649008 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44613
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:12.962366 13090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:12.962631 13090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:12.962806 13090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:12.969928 13090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:12.970041 13090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:12.970227 13090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:54:12.979730 13090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:58355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:12.980958 13090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:12.982348 13090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:54:12.991204 12941 heartbeater.cc:503] Master 127.0.97.62:44613 was elected leader, sending a full tablet report...
W20260927 22:54:12.995247 13095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:12.995290 13096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:12.995319 13098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:12.996235 13090 server_base.cc:1034] running on GCE node
I20260927 22:54:12.996789 13090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:12.997700 13090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:12.998911 13090 hybrid_clock.cc:648] HybridClock initialized: now 1790549652998863 us; error 59 us; skew 500 ppm
I20260927 22:54:12.999210 13090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:13 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:13.007683 13090 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:54:13.009035 13090 webserver.cc:460] Webserver started at http://127.0.97.3:43585/ using document root <none> and password file <none>
I20260927 22:54:13.009717 13090 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:13.009824 13090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:13.010064 13090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:54:13 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:13.025415 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "a19c8498bc3e49dda7b92364ae6d4d5d"
format_stamp: "Formatted at 2026-09-27 22:54:13 on dist-test-slave-df53"
server_key: "ee512c131a1684a9ab5ae06b47105600"
server_key_iv: "54516c13de337d9f0fd0ee014319eee8"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:13.026451 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a19c8498bc3e49dda7b92364ae6d4d5d"
format_stamp: "Formatted at 2026-09-27 22:54:13 on dist-test-slave-df53"
server_key: "ee512c131a1684a9ab5ae06b47105600"
server_key_iv: "54516c13de337d9f0fd0ee014319eee8"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:13.033254 13090 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260927 22:54:13.048105 13105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:13.049711 13090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:13.049892 13090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "a19c8498bc3e49dda7b92364ae6d4d5d"
format_stamp: "Formatted at 2026-09-27 22:54:13 on dist-test-slave-df53"
server_key: "ee512c131a1684a9ab5ae06b47105600"
server_key_iv: "54516c13de337d9f0fd0ee014319eee8"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:13.050100 13090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:13.091094 13090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:13.095650 13090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:13.095978 13090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:13.097150 13090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:13.098996 13090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:13.099084 13090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:13.099267 13090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:13.099346 13090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:13.132035 13090 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33025
I20260927 22:54:13.132071 13217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33025 every 8 connection(s)
I20260927 22:54:13.133293 13090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:13.141407 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 13090
I20260927 22:54:13.141597 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:54:13.142205 388 external_mini_cluster.cc:1428] Setting key c47b0639303cae838170ca416d3a7c2a
Sep 27 22:54:13 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:13.152096 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:13.135867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33969 (local address 127.0.97.62:44613)
0927 22:54:13.136358 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:13.136363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:13.137309 (+ 946us) server_negotiation.cc:407] Connection header received
0927 22:54:13.139070 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:13.139076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:13.139159 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:13.139297 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:13.140620 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.141431 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.143499 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.143954 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.144097 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.147255 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:13.147260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:13.147263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:13.147293 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:13.149713 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:13.150334 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:13.150337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:13.150340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:13.150452 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:13.150803 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:13.150806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:13.150808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:13.151047 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:13.151150 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:13.151701 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 22:54:13.151863 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":223,"threads_started":1}
I20260927 22:54:13.152945 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:13.136444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44613 (local address 127.0.97.3:33969)
0927 22:54:13.137139 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:13.137181 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:13.138816 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:13.139508 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:13.139517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:13.139914 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:13.140461 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:13.140476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.141642 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.141650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:13.143351 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:13.143362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.144174 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.144182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:13.144412 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.145066 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:13.145087 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:13.147067 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:13.149834 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:13.149840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:13.149851 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:13.150227 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:13.150539 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:13.150542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:13.150544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:13.150710 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:13.151150 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:13.151155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:13.151406 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 22:54:13.151606 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 22:54:13.151926 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":303,"threads_started":1}
I20260927 22:54:13.154388 13218 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44613
I20260927 22:54:13.154672 13218 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:13.155442 13218 heartbeater.cc:511] Master 127.0.97.62:44613 requested a full tablet report, sending...
I20260927 22:54:13.156826 12728 ts_manager.cc:194] Registered new tserver with Master: a19c8498bc3e49dda7b92364ae6d4d5d (127.0.97.3:33025)
I20260927 22:54:13.157733 12728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33969
I20260927 22:54:13.169209 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:54:13.170913 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 12808
Sep 27 22:54:13 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:54:13.186439 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:13.171731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.0.97.1:39809)
0927 22:54:13.172330 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:13.172336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:13.172359 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:54:13.172411 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:13.172417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:13.172586 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:13.172709 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:13.174299 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.175263 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.176695 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.177189 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.177281 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.181263 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:13.181273 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:13.181285 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:13.181319 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:13.184152 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:13.184696 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:13.184702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:13.184704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:13.184856 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:13.185193 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:13.185197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:13.185199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:13.185485 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:13.185759 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:13.185998 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 22:54:13.186176 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":319,"threads_started":1}
I20260927 22:54:13.187521 12936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:13.276998 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 12808
I20260927 22:54:13.292053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 12947
Sep 27 22:54:13 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:54:13.307364 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:13.292847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.0.97.2:38923)
0927 22:54:13.293408 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:13.293415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:13.293438 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:54:13.293494 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:13.293501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:13.293773 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:13.293898 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:13.294975 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.295937 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.297143 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.297818 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.297920 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.302168 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:13.302178 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:13.302191 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:13.302219 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:13.304889 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:13.305519 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:13.305526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:13.305528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:13.305698 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:13.306058 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:13.306062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:13.306064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:13.306427 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:13.306645 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:13.306911 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 22:54:13.307089 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":278,"threads_started":1}
I20260927 22:54:13.308529 13079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:13.389163 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 12947
I20260927 22:54:13.404279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 13090
Sep 27 22:54:13 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:54:13.420735 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:13.405269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.0.97.3:33025)
0927 22:54:13.405980 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:13.405987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:13.406016 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:54:13.406075 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:13.406081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:13.406312 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:13.406442 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:13.407756 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.408757 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.410162 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.410731 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.410831 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.415138 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:13.415149 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:13.415163 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:13.415201 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:13.417977 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:13.418738 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:13.418746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:13.418749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:13.418895 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:13.419265 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:13.419268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:13.419271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:13.419595 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:13.419867 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:13.420108 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 22:54:13.420312 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":420,"threads_started":1}
I20260927 22:54:13.422379 13213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:13.504817 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 13090
I20260927 22:54:13.520032 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 12696
I20260927 22:54:13.521256 12758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:13.629493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 12696
I20260927 22:54:13.652136 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44613
--webserver_interface=127.0.97.62
--webserver_port=39651
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44613
--ranger_config_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58355/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58355/kms with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:13.961265 13230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:13.961544 13230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:13.961755 13230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:13.969925 13230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:13.970026 13230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:13.970124 13230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:13.970218 13230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:13.970286 13230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:13.980048 13230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58355/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44613
--ranger_config_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:13.981244 13230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:13.982566 13230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:13.994191 13235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:13.994237 13238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:13.994158 13236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:13.995244 13230 server_base.cc:1034] running on GCE node
I20260927 22:54:13.995966 13230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:13.997292 13230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:13.998533 13230 hybrid_clock.cc:648] HybridClock initialized: now 1790549653998485 us; error 61 us; skew 500 ppm
I20260927 22:54:13.998894 13230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:13 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:14.001988 13230 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:54:14.003346 13230 webserver.cc:460] Webserver started at http://127.0.97.62:39651/ using document root <none> and password file <none>
I20260927 22:54:14.003958 13230 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:14.004105 13230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:54:14 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:14.067555 13230 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:54:14.071080 13246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:14.072757 13230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:54:14.072940 13230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "c16633baddc746c598a2bca105ab3832"
format_stamp: "Formatted at 2026-09-27 22:54:10 on dist-test-slave-df53"
server_key: "0aa352b484968c83b634e97a28ad1b14"
server_key_iv: "000bc98c0e484b332823fbeedfb9933d"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:14.074039 13230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:14.094102 13230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:14.097486 13230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:14.097797 13230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:14.115518 13230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44613
I20260927 22:54:14.115541 13297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44613 every 8 connection(s)
I20260927 22:54:14.116883 13230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:14.122641 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 13230
I20260927 22:54:14.123423 13298 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:54:14.123504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39809
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44613
--webserver_port=44575
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58355/kms with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:54:14.126938 13298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: Bootstrap starting.
I20260927 22:54:14.145144 13298 log.cc:824] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:14.153342 13298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:54:14.153925 13298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: Bootstrap complete.
I20260927 22:54:14.162545 13298 raft_consensus.cc:348] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } }
I20260927 22:54:14.163358 13298 raft_consensus.cc:729] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16633baddc746c598a2bca105ab3832, State: Initialized, Role: FOLLOWER
I20260927 22:54:14.163975 13298 consensus_queue.cc:260] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } }
I20260927 22:54:14.164203 13298 raft_consensus.cc:388] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:14.164325 13298 raft_consensus.cc:482] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:14.164505 13298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:54:14.168069 13298 raft_consensus.cc:504] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } }
I20260927 22:54:14.168516 13298 leader_election.cc:302] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c16633baddc746c598a2bca105ab3832; no voters:
I20260927 22:54:14.169112 13298 leader_election.cc:288] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:54:14.169317 13302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:54:14.170377 13302 raft_consensus.cc:686] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [term 2 LEADER]: Becoming Leader. State: Replica: c16633baddc746c598a2bca105ab3832, State: Running, Role: LEADER
I20260927 22:54:14.170987 13298 sys_catalog.cc:564] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:14.171123 13302 consensus_queue.cc:237] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } }
I20260927 22:54:14.174196 13303 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c16633baddc746c598a2bca105ab3832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } } }
I20260927 22:54:14.174248 13304 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c16633baddc746c598a2bca105ab3832. Latest consensus state: current_term: 2 leader_uuid: "c16633baddc746c598a2bca105ab3832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16633baddc746c598a2bca105ab3832" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44613 } } }
I20260927 22:54:14.174558 13303 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:14.174643 13304 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:14.185475 13310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:14.193061 13310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:14.194905 13310 catalog_manager.cc:1259] Loaded cluster ID: c877cba986784e049f86a5c7f37817e1
I20260927 22:54:14.195082 13310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:14.200047 13310 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:14.201179 13298 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:54:14.201649 13310 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: Loaded TSK: 0
I20260927 22:54:14.203256 13310 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:54:14.548139 13300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:14.548532 13300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:14.548770 13300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:14.560786 13300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:14.560918 13300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:14.561107 13300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:54:14.576613 13300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44575
--tserver_master_addrs=127.0.97.62:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:14.578437 13300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:14.580142 13300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:14.603866 13335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:14.618382 13336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:14.620993 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:14.621497 13300 server_base.cc:1034] running on GCE node
I20260927 22:54:14.622473 13300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:14.637840 13300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:14.645407 13300 hybrid_clock.cc:648] HybridClock initialized: now 1790549654645318 us; error 94 us; skew 500 ppm
I20260927 22:54:14.645866 13300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:14 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:14.650063 13300 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:54:14.686074 13300 webserver.cc:460] Webserver started at http://127.0.97.1:44575/ using document root <none> and password file <none>
I20260927 22:54:14.687222 13300 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:14.687470 13300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:54:14 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:14.723783 13300 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.003s
I20260927 22:54:14.729349 13347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:14.731177 13300 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:54:14.731441 13300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "6d1867f251fc4a8cbc0967d205238415"
format_stamp: "Formatted at 2026-09-27 22:54:11 on dist-test-slave-df53"
server_key: "a29be560fb32646a6f21c60ec7ed5e22"
server_key_iv: "2838bbca1d0f0bc42c9f9eb1642abe88"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:14.732883 13300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:14.779332 13300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:14.784412 13300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:14.784726 13300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:14.785912 13300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:14.787711 13300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:14.787865 13300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:14.788010 13300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:14.788102 13300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:14.839156 13300 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39809
I20260927 22:54:14.839205 13459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39809 every 8 connection(s)
I20260927 22:54:14.840618 13300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:14.850507 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 13300
I20260927 22:54:14.851289 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38923
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44613
--webserver_port=39449
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58355/kms with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:54:14 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:14.869344 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:14.843983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44613 (local address 127.0.97.1:58755)
0927 22:54:14.845394 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:14.845428 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:14.847097 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:14.848462 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:14.848473 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:14.849022 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:14.849524 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:14.849540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.851938 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.851944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:14.854022 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:14.854046 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.855602 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.855609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:14.855865 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:14.857072 (+ 1207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:14.857106 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:14.860013 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:14.864559 (+ 4546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:14.864568 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:14.864582 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:14.865179 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:14.865740 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:14.865745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:14.865749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:14.865922 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:14.866803 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:14.866810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:14.867176 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 22:54:14.867487 (+ 311us) client_negotiation.cc:240] Negotiation successful
0927 22:54:14.868002 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"thread_start_us":1025,"threads_started":1}
I20260927 22:54:14.871719 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:14.843712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58755 (local address 127.0.97.62:44613)
0927 22:54:14.844886 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:14.844900 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:14.845567 (+ 667us) server_negotiation.cc:407] Connection header received
0927 22:54:14.847643 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:14.847655 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:14.847963 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:14.848321 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:14.849747 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.851728 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.854209 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.855359 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.855540 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:14.860348 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:14.860358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:14.860377 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:14.860425 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:14.864385 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:14.865354 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:14.865363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:14.865369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:14.865555 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:14.866048 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:14.866054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:14.866057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:14.866519 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:14.866724 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:14.870148 (+ 3424us) server_negotiation.cc:299] Negotiation successful
0927 22:54:14.870543 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":726,"threads_started":1}
I20260927 22:54:14.875109 13460 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44613
I20260927 22:54:14.875428 13460 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:14.876207 13460 heartbeater.cc:511] Master 127.0.97.62:44613 requested a full tablet report, sending...
W20260927 22:54:14.879182 13460 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44613 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:54:14.879410 13460 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:14.879655 13460 heartbeater.cc:511] Master 127.0.97.62:44613 requested a full tablet report, sending...
I20260927 22:54:14.880666 13460 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:14.880905 13460 heartbeater.cc:511] Master 127.0.97.62:44613 requested a full tablet report, sending...
W20260927 22:54:14.881867 13460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:54:15.274323 13464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:15.274650 13464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:15.274806 13464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:15.282436 13464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:15.282579 13464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:15.282768 13464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:54:15.293232 13464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39449
--tserver_master_addrs=127.0.97.62:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:15.294806 13464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:15.296478 13464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:15.322053 13478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:15.322427 13479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:15.334964 13482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:15.335285 13464 server_base.cc:1034] running on GCE node
I20260927 22:54:15.336098 13464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:15.338207 13464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:15.339578 13464 hybrid_clock.cc:648] HybridClock initialized: now 1790549655339494 us; error 65 us; skew 500 ppm
I20260927 22:54:15.339995 13464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:15 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:15.350313 13464 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:54:15.370023 13464 webserver.cc:460] Webserver started at http://127.0.97.2:39449/ using document root <none> and password file <none>
I20260927 22:54:15.371039 13464 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:15.371209 13464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:54:15 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:15.399618 13464 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:54:15.403285 13491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:15.405061 13464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:54:15.405251 13464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "b9b1045dfc5745139a733f463ba4c020"
format_stamp: "Formatted at 2026-09-27 22:54:12 on dist-test-slave-df53"
server_key: "2ef2e3030dd37531c5c3091c7f9aeeec"
server_key_iv: "02eb01dc06e7db36f43753735a0ba825"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:15.406656 13464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 22:54:15 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:15.438261 13464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:15.443509 13464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:15.443866 13464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:15.449167 13464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:15.453341 13464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:15.453470 13464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:15.453629 13464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:15.453732 13464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 22:54:15.524931 13464 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38923
I20260927 22:54:15.525024 13604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38923 every 8 connection(s)
I20260927 22:54:15.526772 13464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:15.531883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 13464
I20260927 22:54:15.532761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58355/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33025
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44613
--webserver_port=43585
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58355/kms with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:54:15 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:15.567790 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:15.531793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44613 (local address 127.0.97.2:49745)
0927 22:54:15.542577 (+ 10784us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:15.542634 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:15.546159 (+ 3525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:15.547587 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:15.547602 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:15.548319 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:15.549221 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:15.549250 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.550884 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.550891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:15.552975 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:15.553003 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.554654 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.554663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:15.554976 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:15.555781 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:15.555810 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:15.558497 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:15.562704 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:15.562714 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:15.562735 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:15.563386 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:15.563856 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:15.563862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:15.563866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:15.564113 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:15.564873 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:15.564884 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:15.565354 (+ 470us) client_negotiation.cc:769] Sending connection context
0927 22:54:15.565707 (+ 353us) client_negotiation.cc:240] Negotiation successful
0927 22:54:15.566310 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10471,"thread_start_us":10216,"threads_started":1}
I20260927 22:54:15.569322 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:15.530879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49745 (local address 127.0.97.62:44613)
0927 22:54:15.547000 (+ 16121us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:15.547004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:15.547021 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:54:15.547073 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:15.547076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:15.547137 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:15.547268 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:15.549454 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.550664 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.553190 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.554178 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.554315 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:15.558797 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:15.558804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:15.558809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:15.558854 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:15.562524 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:15.563543 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:15.563550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:15.563555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:15.563708 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:15.564273 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:15.564278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:15.564281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:15.564645 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:15.564782 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:15.568673 (+ 3891us) server_negotiation.cc:299] Negotiation successful
0927 22:54:15.568933 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16001,"thread_start_us":15877,"threads_started":1}
I20260927 22:54:15.584580 13605 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44613
I20260927 22:54:15.585011 13605 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:15.586261 13605 heartbeater.cc:511] Master 127.0.97.62:44613 requested a full tablet report, sending...
I20260927 22:54:15.588603 13263 ts_manager.cc:194] Registered new tserver with Master: b9b1045dfc5745139a733f463ba4c020 (127.0.97.2:38923)
I20260927 22:54:15.590824 13263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49745
Sep 27 22:54:15 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:15.882009 13460 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:15.882438 13460 heartbeater.cc:511] Master 127.0.97.62:44613 requested a full tablet report, sending...
I20260927 22:54:15.883701 13263 ts_manager.cc:194] Registered new tserver with Master: 6d1867f251fc4a8cbc0967d205238415 (127.0.97.1:39809)
I20260927 22:54:15.884773 13263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58755
W20260927 22:54:15.959357 13609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:15.959717 13609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:15.959875 13609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:15.969102 13609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:15.969244 13609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:15.969456 13609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:54:15.980433 13609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:58355/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43585
--tserver_master_addrs=127.0.97.62:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:15.981835 13609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:15.983110 13609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:16.000032 13615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:16.000221 13614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:16.000063 13617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:16.001348 13609 server_base.cc:1034] running on GCE node
I20260927 22:54:16.002067 13609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:16.003089 13609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:16.004374 13609 hybrid_clock.cc:648] HybridClock initialized: now 1790549656004299 us; error 70 us; skew 500 ppm
I20260927 22:54:16.004693 13609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:16 dist-test-slave-df53 krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:16.008102 13609 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:54:16.011327 13609 webserver.cc:460] Webserver started at http://127.0.97.3:43585/ using document root <none> and password file <none>
I20260927 22:54:16.012041 13609 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:16.012203 13609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:54:16 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:16.040601 13609 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:54:16.043541 13625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:16.044978 13609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:54:16.045157 13609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "a19c8498bc3e49dda7b92364ae6d4d5d"
format_stamp: "Formatted at 2026-09-27 22:54:13 on dist-test-slave-df53"
server_key: "ee512c131a1684a9ab5ae06b47105600"
server_key_iv: "54516c13de337d9f0fd0ee014319eee8"
server_key_version: "kuduclusterkey@0"
I20260927 22:54:16.046200 13609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:16.070757 13609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:16.075112 13609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:16.075395 13609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:16.076422 13609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:16.078084 13609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:16.078194 13609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:16.078297 13609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:16.078390 13609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:16.141167 13609 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33025
I20260927 22:54:16.142139 13737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33025 every 8 connection(s)
I20260927 22:54:16.143193 13609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:16.153096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 13609
I20260927 22:54:16.198289 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.154203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.0.97.62:44613)
0927 22:54:16.175710 (+ 21507us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.175719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.175739 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:54:16.175796 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.175802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.175890 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.176002 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:16.180395 (+ 4393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.181522 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.184166 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.184785 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.184896 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.190686 (+ 5790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:16.190692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:16.190719 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:16.190767 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:16.195142 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:16.196005 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:16.196011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:16.196014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:16.196155 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:16.196572 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:16.196576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:16.196579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:16.196893 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:16.197154 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.197573 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.197805 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21374,"thread_start_us":21228,"threads_started":1}
Sep 27 22:54:16 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 22:54:16 dist-test-slave-df53 krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 22:54:16.207121 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.147194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33621 (local address 127.0.97.62:44613)
0927 22:54:16.164509 (+ 17315us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.164518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.164543 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:54:16.164621 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.164628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.164746 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.164881 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:16.178082 (+ 13201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.179250 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.181666 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.182304 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.182405 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.200342 (+ 17937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:16.200348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:16.200353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:16.200388 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:16.203479 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:16.204424 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:16.204430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:16.204433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:16.204569 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:16.205116 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:16.205120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:16.205123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:16.205430 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:16.205556 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.206559 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.206755 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17165,"thread_start_us":17044,"threads_started":1}
I20260927 22:54:16.209393 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.146622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44613 (local address 127.0.97.3:33621)
0927 22:54:16.160239 (+ 13617us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.160310 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.163503 (+ 3193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.175753 (+ 12250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.175768 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.176830 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:16.177864 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.177893 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.179450 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.179455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.181431 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.181456 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.186053 (+ 4597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.186060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.186350 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.191571 (+ 5221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:16.191604 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:16.200049 (+ 8445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:16.203624 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:16.203634 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:16.203651 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:16.204271 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:16.204725 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:16.204730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:16.204733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:16.204986 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:16.205675 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:16.205682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:16.206160 (+ 478us) client_negotiation.cc:769] Sending connection context
0927 22:54:16.207519 (+ 1359us) client_negotiation.cc:240] Negotiation successful
0927 22:54:16.208070 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13250,"thread_start_us":8173,"threads_started":1}
I20260927 22:54:16.211228 13738 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44613
I20260927 22:54:16.211573 13738 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:16.212374 13738 heartbeater.cc:511] Master 127.0.97.62:44613 requested a full tablet report, sending...
I20260927 22:54:16.214221 13263 ts_manager.cc:194] Registered new tserver with Master: a19c8498bc3e49dda7b92364ae6d4d5d (127.0.97.3:33025)
I20260927 22:54:16.215241 13263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33621
I20260927 22:54:16.226372 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:54:16.280799 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.256725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.0.97.62:44613)
0927 22:54:16.256907 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.256914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.264144 (+ 7230us) server_negotiation.cc:407] Connection header received
0927 22:54:16.264631 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.264637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.264720 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.264790 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:16.266863 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.267885 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.269838 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.270448 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.270541 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.274046 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:16.274053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:16.274058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:16.274098 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:16.277694 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:16.278576 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:16.278582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:16.278585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:16.278726 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:16.279157 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:16.279162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:16.279165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:16.279505 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:16.279725 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.280241 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.280469 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:54:16.287825 13263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:54:16.299194 13263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:54:16.341110 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.325516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33025 (local address 127.0.0.1:43400)
0927 22:54:16.326226 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.326264 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.326440 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.328747 (+ 2307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.328755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.328795 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:16.329156 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.329170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.332529 (+ 3359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.332536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.335890 (+ 3354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.335907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.340155 (+ 4248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.340162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.340381 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.340446 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 22:54:16.340554 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:54:16.340750 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":850,"threads_started":1}
I20260927 22:54:16.342700 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.326170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.0.97.3:33025)
0927 22:54:16.328084 (+ 1914us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.328093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.328124 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:54:16.328220 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.328228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.328498 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.329387 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:16.329689 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.332355 (+ 2666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.336101 (+ 3746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.337681 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.341877 (+ 4196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.342119 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.342195 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.342353 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1748,"thread_start_us":1611,"threads_started":1}
I20260927 22:54:16.352924 13673 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab01cbc00ea4e609c378d9fe31e9329 (DEFAULT_TABLE table=test-table [id=026d1fb9bd0940d9a6b4fbb229b49bd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:16.354774 13673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab01cbc00ea4e609c378d9fe31e9329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:16.357436 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.328614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39809 (local address 127.0.0.1:56264)
0927 22:54:16.338214 (+ 9600us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.338234 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.338365 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.343725 (+ 5360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.343730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.343756 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:16.344112 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.344125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.347382 (+ 3257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.347387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.349530 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.349546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.356606 (+ 7060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.356614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.356789 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.356825 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:54:16.356922 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:54:16.357063 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9471,"thread_start_us":20197,"threads_started":1}
I20260927 22:54:16.358193 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.329084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.0.97.1:39809)
0927 22:54:16.338883 (+ 9799us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.338893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.338922 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:54:16.342946 (+ 4024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.342967 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.343262 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.343486 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:16.344467 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.347186 (+ 2719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.351031 (+ 3845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.356088 (+ 5057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.356219 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.356461 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.357697 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.357866 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9624,"thread_start_us":16717,"threads_started":1}
I20260927 22:54:16.362145 13395 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab01cbc00ea4e609c378d9fe31e9329 (DEFAULT_TABLE table=test-table [id=026d1fb9bd0940d9a6b4fbb229b49bd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:16.363441 13395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab01cbc00ea4e609c378d9fe31e9329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:16.364095 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.338062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38923 (local address 127.0.0.1:32940)
0927 22:54:16.341607 (+ 3545us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.341626 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.341723 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.353011 (+ 11288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.353016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.353044 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:16.353209 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.353219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.358458 (+ 5239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.358464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.360307 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.360331 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.363328 (+ 2997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.363335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.363529 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.363570 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:54:16.363641 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:54:16.363779 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3437,"mutex_wait_us":513,"thread_start_us":19862,"threads_started":1}
I20260927 22:54:16.365052 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.338411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.0.97.2:38923)
0927 22:54:16.350072 (+ 11661us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.350081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.350110 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:54:16.350206 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.350214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.350493 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.353402 (+ 2909us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:16.353685 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.355275 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.361713 (+ 6438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.363151 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.364313 (+ 1162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.364517 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.364587 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.364746 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11491,"threads_started":1}
I20260927 22:54:16.376194 13539 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab01cbc00ea4e609c378d9fe31e9329 (DEFAULT_TABLE table=test-table [id=026d1fb9bd0940d9a6b4fbb229b49bd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:16.377611 13539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab01cbc00ea4e609c378d9fe31e9329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:16.385037 13760 tablet_bootstrap.cc:492] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415: Bootstrap starting.
I20260927 22:54:16.397130 13760 tablet_bootstrap.cc:654] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:16.398476 13760 log.cc:824] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:16.403049 13763 tablet_bootstrap.cc:492] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: Bootstrap starting.
I20260927 22:54:16.405078 13761 tablet_bootstrap.cc:492] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: Bootstrap starting.
I20260927 22:54:16.408550 13763 tablet_bootstrap.cc:654] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:16.413374 13760 tablet_bootstrap.cc:492] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415: No bootstrap required, opened a new log
I20260927 22:54:16.413709 13761 tablet_bootstrap.cc:654] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:16.413805 13760 ts_tablet_manager.cc:1397] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
I20260927 22:54:16.414898 13761 log.cc:824] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:16.415186 13763 log.cc:824] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:16.427469 13761 tablet_bootstrap.cc:492] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: No bootstrap required, opened a new log
I20260927 22:54:16.428157 13761 ts_tablet_manager.cc:1397] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.005s
I20260927 22:54:16.433936 13760 raft_consensus.cc:348] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } }
I20260927 22:54:16.434423 13760 raft_consensus.cc:374] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:16.434546 13760 raft_consensus.cc:729] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1867f251fc4a8cbc0967d205238415, State: Initialized, Role: FOLLOWER
I20260927 22:54:16.436462 13763 tablet_bootstrap.cc:492] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: No bootstrap required, opened a new log
I20260927 22:54:16.436242 13760 consensus_queue.cc:260] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } }
I20260927 22:54:16.436890 13763 ts_tablet_manager.cc:1397] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.008s
I20260927 22:54:16.446882 13763 raft_consensus.cc:348] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } }
I20260927 22:54:16.447417 13763 raft_consensus.cc:374] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:16.447556 13763 raft_consensus.cc:729] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b1045dfc5745139a733f463ba4c020, State: Initialized, Role: FOLLOWER
I20260927 22:54:16.448333 13763 consensus_queue.cc:260] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } }
I20260927 22:54:16.451121 13460 heartbeater.cc:503] Master 127.0.97.62:44613 was elected leader, sending a full tablet report...
I20260927 22:54:16.451040 13761 raft_consensus.cc:348] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } }
I20260927 22:54:16.451507 13761 raft_consensus.cc:374] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:16.451604 13761 raft_consensus.cc:729] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19c8498bc3e49dda7b92364ae6d4d5d, State: Initialized, Role: FOLLOWER
I20260927 22:54:16.452323 13761 consensus_queue.cc:260] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } }
I20260927 22:54:16.457947 13760 ts_tablet_manager.cc:1428] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415: Time spent starting tablet: real 0.044s user 0.018s sys 0.008s
I20260927 22:54:16.464359 13605 heartbeater.cc:503] Master 127.0.97.62:44613 was elected leader, sending a full tablet report...
I20260927 22:54:16.464483 13738 heartbeater.cc:503] Master 127.0.97.62:44613 was elected leader, sending a full tablet report...
I20260927 22:54:16.465286 13761 ts_tablet_manager.cc:1428] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: Time spent starting tablet: real 0.037s user 0.017s sys 0.008s
I20260927 22:54:16.465301 13763 ts_tablet_manager.cc:1428] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: Time spent starting tablet: real 0.028s user 0.011s sys 0.006s
W20260927 22:54:16.551013 13606 tablet.cc:2369] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:54:16.593293 13461 tablet.cc:2369] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:54:16.667478 13739 tablet.cc:2369] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:54:16.806222 13768 raft_consensus.cc:482] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:54:16.806668 13768 raft_consensus.cc:504] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } }
I20260927 22:54:16.807413 13767 raft_consensus.cc:482] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:54:16.807672 13767 raft_consensus.cc:504] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } }
I20260927 22:54:16.808789 13768 leader_election.cc:288] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a19c8498bc3e49dda7b92364ae6d4d5d (127.0.97.3:33025), 6d1867f251fc4a8cbc0967d205238415 (127.0.97.1:39809)
I20260927 22:54:16.809628 13767 leader_election.cc:288] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a19c8498bc3e49dda7b92364ae6d4d5d (127.0.97.3:33025), b9b1045dfc5745139a733f463ba4c020 (127.0.97.2:38923)
I20260927 22:54:16.819429 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.809099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48255 (local address 127.0.97.3:33025)
0927 22:54:16.809305 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.809311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.809772 (+ 461us) server_negotiation.cc:407] Connection header received
0927 22:54:16.810091 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.810098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.810179 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.810246 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:16.811428 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.812883 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.816351 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.818142 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.818270 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.818417 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.819035 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.819183 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:54:16.820222 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.809009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33025 (local address 127.0.97.2:48255)
0927 22:54:16.809679 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.809697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.809905 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.810588 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.810595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.810632 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:16.811195 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.811211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.813184 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.813192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.816135 (+ 2943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.816155 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.818632 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.818641 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.818845 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.818884 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:54:16.819665 (+ 781us) client_negotiation.cc:240] Negotiation successful
0927 22:54:16.819831 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":343,"threads_started":1}
I20260927 22:54:16.821384 13693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab01cbc00ea4e609c378d9fe31e9329" candidate_uuid: "b9b1045dfc5745139a733f463ba4c020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" is_pre_election: true
I20260927 22:54:16.821962 13693 raft_consensus.cc:2454] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b1045dfc5745139a733f463ba4c020 in term 0.
I20260927 22:54:16.822757 13494 leader_election.cc:302] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a19c8498bc3e49dda7b92364ae6d4d5d, b9b1045dfc5745139a733f463ba4c020; no voters:
I20260927 22:54:16.823249 13768 raft_consensus.cc:2781] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:54:16.823380 13768 raft_consensus.cc:482] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:54:16.823475 13768 raft_consensus.cc:3037] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:16.826474 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.810245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34857 (local address 127.0.97.1:39809)
0927 22:54:16.810867 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.810875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.819329 (+ 8454us) server_negotiation.cc:407] Connection header received
0927 22:54:16.819577 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.819584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.819650 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.819708 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:16.820529 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.821677 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.824393 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.825488 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.825664 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.825766 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.826088 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.826219 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":464}
I20260927 22:54:16.826543 13768 raft_consensus.cc:504] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } }
I20260927 22:54:16.827334 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.809767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39809 (local address 127.0.97.2:34857)
0927 22:54:16.819231 (+ 9464us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.819248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.819420 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.819910 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.819917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.819950 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:16.820348 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.820363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.821860 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.821868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.824203 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.824218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.825725 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.825731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.825899 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.825935 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:54:16.826867 (+ 932us) client_negotiation.cc:240] Negotiation successful
0927 22:54:16.827000 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9349,"thread_start_us":9453,"threads_started":1}
I20260927 22:54:16.827397 13768 leader_election.cc:288] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [CANDIDATE]: Term 1 election: Requested vote from peers a19c8498bc3e49dda7b92364ae6d4d5d (127.0.97.3:33025), 6d1867f251fc4a8cbc0967d205238415 (127.0.97.1:39809)
I20260927 22:54:16.827643 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.809710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33025 (local address 127.0.97.1:56497)
0927 22:54:16.816417 (+ 6707us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.816441 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.816597 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.819147 (+ 2550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.819152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.819179 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:16.819869 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.819885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.821927 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.821934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.824974 (+ 3040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.824987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.826780 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.826786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.826978 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.827011 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:54:16.827105 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:54:16.827233 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6558,"mutex_wait_us":1,"thread_start_us":6367,"threads_started":1}
I20260927 22:54:16.828255 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.810031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56497 (local address 127.0.97.3:33025)
0927 22:54:16.818865 (+ 8834us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.818870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.818886 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:54:16.818951 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.818957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.819033 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.819119 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:16.820255 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.821787 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.825133 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.826379 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.826504 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.826621 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.827765 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.827903 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8733,"thread_start_us":8602,"threads_started":1}
I20260927 22:54:16.828235 13693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab01cbc00ea4e609c378d9fe31e9329" candidate_uuid: "b9b1045dfc5745139a733f463ba4c020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d"
I20260927 22:54:16.828608 13693 raft_consensus.cc:3037] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:16.829041 13692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab01cbc00ea4e609c378d9fe31e9329" candidate_uuid: "6d1867f251fc4a8cbc0967d205238415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" is_pre_election: true
I20260927 22:54:16.829285 13414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab01cbc00ea4e609c378d9fe31e9329" candidate_uuid: "b9b1045dfc5745139a733f463ba4c020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1867f251fc4a8cbc0967d205238415"
I20260927 22:54:16.829695 13414 raft_consensus.cc:3037] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:16.829988 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.819466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33161 (local address 127.0.97.2:38923)
0927 22:54:16.819624 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.819630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.820215 (+ 585us) server_negotiation.cc:407] Connection header received
0927 22:54:16.820460 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.820465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.820555 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.820619 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:16.822326 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.823495 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.827596 (+ 4101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.828837 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.828966 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.829046 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.829632 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.829767 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 22:54:16.830736 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.810436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38923 (local address 127.0.97.1:33161)
0927 22:54:16.820117 (+ 9681us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.820135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.820309 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.820817 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.820821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.820848 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:16.822092 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.822107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.823668 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.823675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.827395 (+ 3720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.827415 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.829197 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.829203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.829399 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.829443 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:54:16.830122 (+ 679us) client_negotiation.cc:240] Negotiation successful
0927 22:54:16.830282 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9557,"mutex_wait_us":789,"thread_start_us":9536,"threads_started":1}
I20260927 22:54:16.830566 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab01cbc00ea4e609c378d9fe31e9329" candidate_uuid: "b9b1045dfc5745139a733f463ba4c020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1867f251fc4a8cbc0967d205238415" is_pre_election: true
I20260927 22:54:16.831692 13560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab01cbc00ea4e609c378d9fe31e9329" candidate_uuid: "6d1867f251fc4a8cbc0967d205238415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b1045dfc5745139a733f463ba4c020" is_pre_election: true
I20260927 22:54:16.832149 13560 raft_consensus.cc:2379] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d1867f251fc4a8cbc0967d205238415 in current term 1: Already voted for candidate b9b1045dfc5745139a733f463ba4c020 in this term.
I20260927 22:54:16.832731 13693 raft_consensus.cc:2454] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b1045dfc5745139a733f463ba4c020 in term 1.
I20260927 22:54:16.832844 13349 leader_election.cc:302] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d1867f251fc4a8cbc0967d205238415; no voters: a19c8498bc3e49dda7b92364ae6d4d5d, b9b1045dfc5745139a733f463ba4c020
I20260927 22:54:16.833034 13414 raft_consensus.cc:2454] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b1045dfc5745139a733f463ba4c020 in term 1.
I20260927 22:54:16.833233 13767 raft_consensus.cc:2726] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:54:16.833258 13494 leader_election.cc:302] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a19c8498bc3e49dda7b92364ae6d4d5d, b9b1045dfc5745139a733f463ba4c020; no voters:
I20260927 22:54:16.833547 13768 raft_consensus.cc:2781] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:16.834049 13768 raft_consensus.cc:686] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 1 LEADER]: Becoming Leader. State: Replica: b9b1045dfc5745139a733f463ba4c020, State: Running, Role: LEADER
I20260927 22:54:16.834481 13768 consensus_queue.cc:237] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } }
I20260927 22:54:16.839090 13263 catalog_manager.cc:5495] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9b1045dfc5745139a733f463ba4c020 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b9b1045dfc5745139a733f463ba4c020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b1045dfc5745139a733f463ba4c020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38923 } health_report { overall_health: HEALTHY } } }
I20260927 22:54:16.896710 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.891317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.0.97.2:38923)
0927 22:54:16.891483 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.891487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.891911 (+ 424us) server_negotiation.cc:407] Connection header received
0927 22:54:16.892066 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.892070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.892154 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.892282 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:54:16.892617 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.893318 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.894710 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.895142 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.895236 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.895704 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:54:16.896209 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:54:16.896339 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.896428 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.896494 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:54:16.904512 13693 raft_consensus.cc:1261] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d [term 1 FOLLOWER]: Refusing update from remote peer b9b1045dfc5745139a733f463ba4c020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:16.904585 13414 raft_consensus.cc:1261] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 1 FOLLOWER]: Refusing update from remote peer b9b1045dfc5745139a733f463ba4c020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:16.905427 13781 consensus_queue.cc:1035] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a19c8498bc3e49dda7b92364ae6d4d5d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:16.905817 13768 consensus_queue.cc:1035] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1867f251fc4a8cbc0967d205238415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:16.916525 13788 mvcc.cc:205] Tried to move back new op lower bound from 7334091394671677440 to 7334091394397507584. Current Snapshot: MvccSnapshot[applied={T|T < 7334091394671677440}]
I20260927 22:54:16.922004 13790 mvcc.cc:205] Tried to move back new op lower bound from 7334091394671677440 to 7334091394397507584. Current Snapshot: MvccSnapshot[applied={T|T < 7334091394671677440}]
I20260927 22:54:16.931969 13263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:54:16.932241 13263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:54:16.935643 13263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c16633baddc746c598a2bca105ab3832: Sending DeleteTablet for 3 replicas of tablet 8ab01cbc00ea4e609c378d9fe31e9329
I20260927 22:54:16.936640 13673 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab01cbc00ea4e609c378d9fe31e9329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43400
I20260927 22:54:16.936738 13395 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab01cbc00ea4e609c378d9fe31e9329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56264
I20260927 22:54:16.936749 13539 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab01cbc00ea4e609c378d9fe31e9329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32940
I20260927 22:54:16.939440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 13300
I20260927 22:54:16.951330 13798 tablet_replica.cc:330] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: stopping tablet replica
I20260927 22:54:16.951851 13798 raft_consensus.cc:2229] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:54:16.952400 13798 raft_consensus.cc:2258] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:16.956197 13798 ts_tablet_manager.cc:1905] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:16.957542 13800 tablet_replica.cc:330] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: stopping tablet replica
I20260927 22:54:16.958050 13800 raft_consensus.cc:2229] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:54:16.958726 13800 raft_consensus.cc:2258] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:16.960768 13800 ts_tablet_manager.cc:1905] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:16.966071 13799 tablet_replica.cc:330] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415: stopping tablet replica
I20260927 22:54:16.966521 13799 raft_consensus.cc:2229] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:54:16.967010 13799 raft_consensus.cc:2258] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:16.968914 13799 ts_tablet_manager.cc:1905] T 8ab01cbc00ea4e609c378d9fe31e9329 P 6d1867f251fc4a8cbc0967d205238415: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:16.972844 13798 ts_tablet_manager.cc:1918] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:16.973083 13798 log.cc:1196] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/8ab01cbc00ea4e609c378d9fe31e9329
I20260927 22:54:16.973687 13798 ts_tablet_manager.cc:1939] T 8ab01cbc00ea4e609c378d9fe31e9329 P a19c8498bc3e49dda7b92364ae6d4d5d: Deleting consensus metadata
I20260927 22:54:16.976421 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.940316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.0.97.1:39809)
0927 22:54:16.940511 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.940518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.956937 (+ 16419us) server_negotiation.cc:407] Connection header received
0927 22:54:16.957165 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.957175 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.957313 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.957439 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:16.959859 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.961421 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.964080 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.965049 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.965175 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.968791 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:16.968804 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:16.968821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:16.968864 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:16.972819 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:16.973801 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:16.973808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:16.973811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:16.973994 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:16.974471 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:16.974475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:16.974478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:16.974887 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:16.975178 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:16.975818 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 22:54:16.976028 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:54:16.976541 13249 catalog_manager.cc:4841] TS a19c8498bc3e49dda7b92364ae6d4d5d (127.0.97.3:33025): tablet 8ab01cbc00ea4e609c378d9fe31e9329 (table test-table [id=026d1fb9bd0940d9a6b4fbb229b49bd6]) successfully deleted
I20260927 22:54:16.977065 13455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:16.979116 13800 ts_tablet_manager.cc:1918] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:16.979535 13800 log.cc:1196] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/8ab01cbc00ea4e609c378d9fe31e9329
I20260927 22:54:16.980082 13800 ts_tablet_manager.cc:1939] T 8ab01cbc00ea4e609c378d9fe31e9329 P b9b1045dfc5745139a733f463ba4c020: Deleting consensus metadata
I20260927 22:54:16.981956 13248 catalog_manager.cc:4841] TS b9b1045dfc5745139a733f463ba4c020 (127.0.97.2:38923): tablet 8ab01cbc00ea4e609c378d9fe31e9329 (table test-table [id=026d1fb9bd0940d9a6b4fbb229b49bd6]) successfully deleted
I20260927 22:54:17.068212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 13300
W20260927 22:54:17.090728 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39809 (error 108)
I20260927 22:54:17.091192 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 13464
I20260927 22:54:17.094362 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.092614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39809 (local address 127.0.0.1:56274)
0927 22:54:17.093997 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:17.094134 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1266,"thread_start_us":1152,"threads_started":1}
W20260927 22:54:17.094756 13249 catalog_manager.cc:4584] TS 6d1867f251fc4a8cbc0967d205238415 (127.0.97.1:39809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ab01cbc00ea4e609c378d9fe31e9329: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39809: connect: Connection refused (error 111)
I20260927 22:54:17.106472 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.092051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.0.97.2:38923)
0927 22:54:17.092267 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:17.092276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:17.092451 (+ 175us) server_negotiation.cc:407] Connection header received
0927 22:54:17.092656 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:17.092662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:17.092760 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:17.092874 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:17.094670 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.095808 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.097115 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.097822 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.097954 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:17.100420 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:17.100433 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:17.100447 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:17.100487 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:17.103533 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:17.104246 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:17.104253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:17.104255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:17.104456 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:17.104902 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:17.104906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:17.104908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:17.105308 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:17.105528 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:17.105929 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 22:54:17.106091 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:54:17.107206 13600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:17.148284 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.147807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39809 (local address 127.0.0.1:56284)
0927 22:54:17.148029 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:17.148136 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 22:54:17.149387 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.148957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39809 (local address 127.0.0.1:56296)
0927 22:54:17.149132 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:17.149223 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:54:17.210055 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.209500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39809 (local address 127.0.0.1:56310)
0927 22:54:17.209770 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:17.209879 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 22:54:17.211211 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.210773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39809 (local address 127.0.0.1:56316)
0927 22:54:17.210960 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:17.211045 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:54:17.221025 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 13464
I20260927 22:54:17.238914 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 13609
I20260927 22:54:17.252437 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.239647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.0.97.3:33025)
0927 22:54:17.239914 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:17.239920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:17.239939 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:54:17.240148 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:17.240152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:17.240236 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:17.240347 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:17.241617 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.242333 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.243607 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.244012 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.244149 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:17.246609 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:17.246621 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:17.246637 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:17.246667 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:17.249577 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:17.250365 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:17.250373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:17.250375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:17.250527 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:17.250936 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:17.250942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:17.250945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:17.251276 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:17.251596 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:17.251886 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 22:54:17.252110 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260927 22:54:17.253077 13733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:17.291733 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.291279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39809 (local address 127.0.0.1:56332)
0927 22:54:17.291514 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:17.291596 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 22:54:17.292761 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.292350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39809 (local address 127.0.0.1:56340)
0927 22:54:17.292547 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:17.292631 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 22:54:17.341089 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 13609
I20260927 22:54:17.357076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 13230
I20260927 22:54:17.358317 13293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:17.469413 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.468894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39809 (local address 127.0.0.1:56352)
0927 22:54:17.469086 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:17.469244 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:54:17.470012 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 13230
I20260927 22:54:17.494359 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:54:17.829200 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:54:17.829393 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:54:18.174312 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:54:18.174 UTC [12282] LOG: received smart shutdown request
2026-09-27 22:54:18.177 UTC [12282] LOG: background worker "logical replication launcher" (PID 12291) exited with exit code 1
2026-09-27 22:54:18.178 UTC [12286] LOG: shutting down
2026-09-27 22:54:18.335 UTC [12282] LOG: database system is shut down
2026-09-27T22:54:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50263 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:54:18 dist-test-slave-df53 krb5kdc[13816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:54:18 dist-test-slave-df53 krb5kdc[13816](info): set up 2 sockets
Sep 27 22:54:18 dist-test-slave-df53 krb5kdc[13816](info): commencing operation
krb5kdc: starting...
W20260927 22:54:19.554049 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:54:19 dist-test-slave-df53 krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:54:19.780834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43817
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43817,127.0.97.61:33137,127.0.97.60:36387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:20.100971 13829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:20.101294 13829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:20.101459 13829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:20.108645 13829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:20.108790 13829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:20.108896 13829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:20.108999 13829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:20.109078 13829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:20.118593 13829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43817,127.0.97.61:33137,127.0.97.60:36387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:20.119840 13829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:20.121104 13829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:20.132599 13834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:20.132578 13837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:20.132534 13835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:20.133462 13829 server_base.cc:1034] running on GCE node
I20260927 22:54:20.134227 13829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:20.135481 13829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:20.136732 13829 hybrid_clock.cc:648] HybridClock initialized: now 1790549660136676 us; error 35 us; skew 500 ppm
I20260927 22:54:20.137096 13829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:20 dist-test-slave-df53 krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:20.140401 13829 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:54:20.141870 13829 webserver.cc:460] Webserver started at http://127.0.97.62:46621/ using document root <none> and password file <none>
I20260927 22:54:20.142496 13829 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:20.142590 13829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:20.142819 13829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:20.146127 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "44ca7ec6e03f42bbb0d8f62af0efa216"
format_stamp: "Formatted at 2026-09-27 22:54:20 on dist-test-slave-df53"
server_key: "71773b6c0eacf81e5ecf030e8f91c270"
server_key_iv: "140b3ecbcfb53c4e32f367ebefad57f0"
server_key_version: "encryptionkey@0"
I20260927 22:54:20.146900 13829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "44ca7ec6e03f42bbb0d8f62af0efa216"
format_stamp: "Formatted at 2026-09-27 22:54:20 on dist-test-slave-df53"
server_key: "71773b6c0eacf81e5ecf030e8f91c270"
server_key_iv: "140b3ecbcfb53c4e32f367ebefad57f0"
server_key_version: "encryptionkey@0"
I20260927 22:54:20.151872 13829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:54:20.155193 13844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:20.156461 13829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:54:20.156632 13829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "44ca7ec6e03f42bbb0d8f62af0efa216"
format_stamp: "Formatted at 2026-09-27 22:54:20 on dist-test-slave-df53"
server_key: "71773b6c0eacf81e5ecf030e8f91c270"
server_key_iv: "140b3ecbcfb53c4e32f367ebefad57f0"
server_key_version: "encryptionkey@0"
I20260927 22:54:20.156795 13829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:20.191092 13829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:20.195318 13829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:20.195623 13829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:20.214843 13829 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43817
I20260927 22:54:20.214919 13895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43817 every 8 connection(s)
I20260927 22:54:20.216220 13829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:20.219753 13896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:20.221577 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 13829
I20260927 22:54:20.221854 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:54:20.222486 388 external_mini_cluster.cc:1428] Setting key 5b5d11462486d23474e52924a5bbe85a
I20260927 22:54:20.226058 13896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:54:20 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 22:54:20.248898 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.228921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43817 (local address 127.0.0.1:45316)
0927 22:54:20.229881 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.229933 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:20.231602 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:20.232487 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:20.232492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:20.233026 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:20.233350 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:20.233366 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.235678 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.235682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:20.236728 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:20.236739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.238138 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.238143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:20.238335 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.239071 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:20.239092 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:20.241453 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:20.244948 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:20.244957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:20.244966 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:20.245403 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:20.245948 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:20.245952 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:20.245954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:20.246097 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:20.247099 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:20.247105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:20.247293 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 22:54:20.247503 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 22:54:20.247860 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":431,"threads_started":1}
I20260927 22:54:20.248939 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.228927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45316 (local address 127.0.97.62:43817)
0927 22:54:20.230021 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:20.230031 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:20.230080 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:54:20.231954 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:20.231965 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:20.232240 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:20.232461 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:20.233539 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.235516 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.236870 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.237920 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.238115 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.241729 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:20.241742 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:20.241762 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:20.241813 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:20.244817 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:20.245550 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:20.245557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:20.245561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:20.245795 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:20.246215 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:20.246242 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:20.246248 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:20.246933 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:20.247135 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:20.247672 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 22:54:20.248060 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":871,"thread_start_us":633,"threads_started":1}
I20260927 22:54:20.252004 13896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:54:20.253153 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.252735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35828)
0927 22:54:20.252947 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.253051 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 22:54:20.253460 13845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
I20260927 22:54:20.255101 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.254651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35830)
0927 22:54:20.254833 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.254932 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 22:54:20.255327 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33137: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 22:54:20.273881 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33137
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43817,127.0.97.61:33137,127.0.97.60:36387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:54:20.304687 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } attempt: 1
I20260927 22:54:20.305936 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.305400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35838)
0927 22:54:20.305640 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.305750 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:54:20.306945 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.306559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35846)
0927 22:54:20.306701 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.306790 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:54:20.307147 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33137: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
I20260927 22:54:20.375423 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } attempt: 2
I20260927 22:54:20.376797 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.376110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35852)
0927 22:54:20.376434 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.376572 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 22:54:20.377954 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.377487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35854)
0927 22:54:20.377675 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.377777 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:54:20.378154 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33137: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
I20260927 22:54:20.469420 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } attempt: 3
I20260927 22:54:20.470782 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.470169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35864)
0927 22:54:20.470439 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.470558 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 22:54:20.472028 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.471523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35880)
0927 22:54:20.471712 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.471821 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:54:20.472235 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33137: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
W20260927 22:54:20.594434 13903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:20.594692 13903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:20.594892 13903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:20.602108 13903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:20.602236 13903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:20.602322 13903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:20.602386 13903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:20.602429 13903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:20.611677 13903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43817,127.0.97.61:33137,127.0.97.60:36387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:20.612884 13903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:20.614063 13903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:54:20.615489 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } attempt: 4
I20260927 22:54:20.616631 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.616120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35892)
0927 22:54:20.616400 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.616511 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 22:54:20.617669 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.617329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35896)
0927 22:54:20.617451 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.617534 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 22:54:20.617882 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33137: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33137: connect: Connection refused (error 111)
W20260927 22:54:20.625763 13911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:20.625883 13908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:20.625734 13909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:20.626477 13903 server_base.cc:1034] running on GCE node
I20260927 22:54:20.627189 13903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:20.628383 13903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:20.629657 13903 hybrid_clock.cc:648] HybridClock initialized: now 1790549660629561 us; error 58 us; skew 500 ppm
I20260927 22:54:20.629982 13903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:20 dist-test-slave-df53 krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:20.638370 13903 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 22:54:20.639722 13903 webserver.cc:460] Webserver started at http://127.0.97.61:41627/ using document root <none> and password file <none>
I20260927 22:54:20.640365 13903 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:20.640503 13903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:20.640794 13903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:20.643643 13903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/data/instance:
uuid: "4c2467608db74d41ba77e5f6da5c6dba"
format_stamp: "Formatted at 2026-09-27 22:54:20 on dist-test-slave-df53"
server_key: "500b57b34c1927fed58636b23bed7443"
server_key_iv: "36cc2d277b6ed4c952aac9f1302fbb34"
server_key_version: "encryptionkey@0"
I20260927 22:54:20.644387 13903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/wal/instance:
uuid: "4c2467608db74d41ba77e5f6da5c6dba"
format_stamp: "Formatted at 2026-09-27 22:54:20 on dist-test-slave-df53"
server_key: "500b57b34c1927fed58636b23bed7443"
server_key_iv: "36cc2d277b6ed4c952aac9f1302fbb34"
server_key_version: "encryptionkey@0"
I20260927 22:54:20.649047 13903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:54:20.652279 13918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:20.653534 13903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:54:20.653738 13903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/wal
uuid: "4c2467608db74d41ba77e5f6da5c6dba"
format_stamp: "Formatted at 2026-09-27 22:54:20 on dist-test-slave-df53"
server_key: "500b57b34c1927fed58636b23bed7443"
server_key_iv: "36cc2d277b6ed4c952aac9f1302fbb34"
server_key_version: "encryptionkey@0"
I20260927 22:54:20.653959 13903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:20.669340 13903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:20.672976 13903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:20.673277 13903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:20.691025 13903 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33137
I20260927 22:54:20.691046 13969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33137 every 8 connection(s)
I20260927 22:54:20.692276 13903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:54:20.693773 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 13903
I20260927 22:54:20.693969 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-1/wal/instance
I20260927 22:54:20.694407 388 external_mini_cluster.cc:1428] Setting key 7a217d9966330dd4ffac1c9811c75e69
I20260927 22:54:20.695920 13970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:20.703012 13970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 22:54:20 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:20.725737 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.706218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.0.97.62:43817)
0927 22:54:20.706440 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:20.706446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:20.707609 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 22:54:20.709379 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:20.709388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:20.709481 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:20.709626 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:20.711007 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.712206 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.713738 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.714509 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.714659 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.718859 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:20.718869 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:20.718877 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:20.718924 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:20.722596 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:20.723285 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:20.723294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:20.723300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:20.723445 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:20.723830 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:20.723838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:20.723844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:20.724231 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:20.724398 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:20.725270 (+ 872us) server_negotiation.cc:299] Negotiation successful
0927 22:54:20.725422 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:54:20.727031 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.706655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43817 (local address 127.0.0.1:45328)
0927 22:54:20.707430 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.707478 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:20.709116 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:20.709830 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:20.709837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:20.710315 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:20.710839 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:20.710856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.712376 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.712380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:20.713545 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:20.713557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.714709 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.714714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:20.714904 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.715603 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:20.715624 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:20.718626 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:20.722736 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:20.722742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:20.722752 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:20.723146 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:20.723547 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:20.723550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:20.723553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:20.723711 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:20.724378 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:20.724389 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:20.724864 (+ 475us) client_negotiation.cc:769] Sending connection context
0927 22:54:20.725176 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 22:54:20.725729 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":357,"threads_started":1}
I20260927 22:54:20.728641 13970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:54:20 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:54:20.742908 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.729356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35904)
0927 22:54:20.729539 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.729560 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:20.729685 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:20.730595 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:20.730600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:20.731025 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:20.731138 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:20.731146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.732450 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.732456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:20.733244 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:20.733254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.734109 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.734115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:20.734358 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.734882 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:20.734896 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:20.736933 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:20.740541 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:20.740544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:20.740546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:20.740906 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:20.741288 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:20.741290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:20.741293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:20.741410 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:20.742049 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:20.742053 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:20.742171 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 22:54:20.742280 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:54:20.742581 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":21}
I20260927 22:54:20.743076 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.729610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.0.97.61:33137)
0927 22:54:20.730245 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:20.730250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:20.730274 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:54:20.730311 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:20.730316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:20.730457 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:20.730608 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:20.731284 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.732307 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.733379 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.733977 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.734109 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.737199 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:20.737216 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:20.737226 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:20.737281 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:20.740421 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:20.741030 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:20.741042 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:20.741048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:20.741181 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:20.741498 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:20.741501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:20.741503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:20.741938 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:20.742091 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:20.742393 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 22:54:20.742672 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":505,"thread_start_us":389,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 22:54:20.745051 13970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:54:20.746016 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.745647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39038)
0927 22:54:20.745789 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.745893 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 22:54:20.746074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36387
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43817,127.0.97.61:33137,127.0.97.60:36387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:20.746321 13919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
I20260927 22:54:20.747786 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.747467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39052)
0927 22:54:20.747626 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.747701 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 22:54:20.748071 13970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
I20260927 22:54:20.797385 13970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } attempt: 1
I20260927 22:54:20.798669 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.798131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39056)
0927 22:54:20.798354 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.798485 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:54:20.799698 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.799306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39066)
0927 22:54:20.799487 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.799574 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 22:54:20.799961 13970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
I20260927 22:54:20.868193 13970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } attempt: 2
I20260927 22:54:20.869464 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.868998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39070)
0927 22:54:20.869211 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.869313 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 22:54:20.870464 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.870154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39086)
0927 22:54:20.870283 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.870351 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:54:20.870699 13970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
I20260927 22:54:20.917121 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } attempt: 5
Sep 27 22:54:20 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:54:20.930235 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.917884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35918)
0927 22:54:20.918100 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.918121 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:20.918261 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:20.918658 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:20.918663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:20.919127 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:20.919308 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:20.919317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.920285 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.920292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:20.921184 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:20.921195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.922036 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.922042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:20.922248 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.922876 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:20.922894 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:20.924833 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:20.927845 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:20.927852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:20.927856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:20.928219 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:20.928575 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:20.928581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:20.928584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:20.928718 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:20.929258 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:20.929262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:20.929423 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 22:54:20.929565 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:54:20.929870 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:54:20.930235 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.917988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.0.97.61:33137)
0927 22:54:20.918148 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:20.918155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:20.918182 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:54:20.918444 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:20.918448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:20.918511 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:20.918626 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:20.919475 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.920137 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.921337 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.921891 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.922026 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.925020 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:20.925024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:20.925028 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:20.925059 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:20.927696 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:20.928343 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:20.928355 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:20.928358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:20.928466 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:20.928829 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:20.928836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:20.928839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:20.929130 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:20.929288 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:20.929697 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 22:54:20.929907 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:54:20.931705 13896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:54:20.932684 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.932318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39102)
0927 22:54:20.932470 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.932552 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 22:54:20.933657 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.933294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39112)
0927 22:54:20.933433 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.933513 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:54:20.933897 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
I20260927 22:54:20.961949 13970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } attempt: 3
I20260927 22:54:20.963028 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.962573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39122)
0927 22:54:20.962794 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.962876 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:54:20.964031 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.963656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39138)
0927 22:54:20.963852 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.963927 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260927 22:54:20.964227 13970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
I20260927 22:54:20.985127 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } attempt: 1
I20260927 22:54:20.986203 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.985793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39152)
0927 22:54:20.985971 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.986057 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:54:20.987192 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.986886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39168)
0927 22:54:20.987028 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.987087 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:54:20.987406 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
I20260927 22:54:21.055617 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } attempt: 2
I20260927 22:54:21.056695 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.056225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39178)
0927 22:54:21.056446 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.056540 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 22:54:21.057809 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.057469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39182)
0927 22:54:21.057642 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.057703 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 22:54:21.058065 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
W20260927 22:54:21.071462 13977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:21.071713 13977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:21.071863 13977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:21.078946 13977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:21.079035 13977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:21.079130 13977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:21.079227 13977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:21.079288 13977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:21.088934 13977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43817,127.0.97.61:33137,127.0.97.60:36387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:21.090229 13977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:21.091480 13977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:21.102792 13982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:21.102747 13985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:21.102834 13983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:21.104023 13977 server_base.cc:1034] running on GCE node
I20260927 22:54:21.104741 13977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:21.106065 13977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:21.107292 13977 hybrid_clock.cc:648] HybridClock initialized: now 1790549661107251 us; error 48 us; skew 500 ppm
I20260927 22:54:21.107448 13970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } attempt: 4
I20260927 22:54:21.107657 13977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:54:21.108537 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.108087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39184)
0927 22:54:21.108299 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.108391 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:54:21.109552 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.109245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39186)
0927 22:54:21.109379 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.109445 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:54:21.109838 13970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Sep 27 22:54:21 dist-test-slave-df53 krb5kdc[13816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:21.111125 13977 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 22:54:21.112490 13977 webserver.cc:460] Webserver started at http://127.0.97.60:36115/ using document root <none> and password file <none>
I20260927 22:54:21.113155 13977 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:21.113298 13977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:21.113626 13977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:21.116423 13977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/data/instance:
uuid: "a67f92ebfbd74774b28a0f0610283bc4"
format_stamp: "Formatted at 2026-09-27 22:54:21 on dist-test-slave-df53"
server_key: "b19814382349bf87537d8c7424fca3dd"
server_key_iv: "263e75abc4db47d30fca15106e6970ad"
server_key_version: "encryptionkey@0"
I20260927 22:54:21.117187 13977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/wal/instance:
uuid: "a67f92ebfbd74774b28a0f0610283bc4"
format_stamp: "Formatted at 2026-09-27 22:54:21 on dist-test-slave-df53"
server_key: "b19814382349bf87537d8c7424fca3dd"
server_key_iv: "263e75abc4db47d30fca15106e6970ad"
server_key_version: "encryptionkey@0"
I20260927 22:54:21.122114 13977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:54:21.125306 13992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:21.126632 13977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:21.126856 13977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/wal
uuid: "a67f92ebfbd74774b28a0f0610283bc4"
format_stamp: "Formatted at 2026-09-27 22:54:21 on dist-test-slave-df53"
server_key: "b19814382349bf87537d8c7424fca3dd"
server_key_iv: "263e75abc4db47d30fca15106e6970ad"
server_key_version: "encryptionkey@0"
I20260927 22:54:21.127066 13977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:21.164258 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } attempt: 3
I20260927 22:54:21.165305 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.164839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39196)
0927 22:54:21.165060 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.165148 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 22:54:21.166297 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.165954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39206)
0927 22:54:21.166114 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.166193 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:54:21.166539 13896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36387: connect: Connection refused (error 111)
I20260927 22:54:21.176528 13977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:21.180436 13977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:21.180707 13977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:21.198649 13977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36387
I20260927 22:54:21.198735 14043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36387 every 8 connection(s)
I20260927 22:54:21.199949 13977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:54:21.203397 14044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:21.206439 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 13977
I20260927 22:54:21.206619 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549465302610-388-0/minicluster-data/master-2/wal/instance
I20260927 22:54:21.207041 388 external_mini_cluster.cc:1428] Setting key 9bb23e12096395ad7957a65e0ed689f7
I20260927 22:54:21.210425 14044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:54:21 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:54:21 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:21.229240 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.213931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.0.97.62:43817)
0927 22:54:21.214170 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.214176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.214874 (+ 698us) server_negotiation.cc:407] Connection header received
0927 22:54:21.215166 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.215175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.215237 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.215352 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.216587 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.217670 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.219366 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.220034 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.220162 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.223737 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.223743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.223748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.223784 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.226505 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.227125 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.227131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.227134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.227263 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.227644 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.227650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.227656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.227916 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.228028 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.228636 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.228859 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:54:21.230290 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.213893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43817 (local address 127.0.0.1:45340)
0927 22:54:21.214697 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.214739 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:21.215002 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:21.215418 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:21.215424 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:21.216084 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:21.216428 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.216440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.217836 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.217841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.219164 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.219179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.220265 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.220273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.220502 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.221388 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:21.221417 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:21.223548 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:21.226643 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:21.226651 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:21.226662 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:21.227012 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:21.227366 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:21.227373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:21.227378 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:21.227536 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:21.228041 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:21.228047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:21.228276 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 22:54:21.228524 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 22:54:21.228954 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":2,"thread_start_us":468,"threads_started":1}
I20260927 22:54:21.230340 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.208870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.0.97.60:36387)
0927 22:54:21.209690 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.209701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.209739 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:54:21.211305 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.211326 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.211689 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.212025 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.213803 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.215846 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.217475 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.218434 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.218606 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.223533 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.223542 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.223555 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.223588 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.226497 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.227126 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.227133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.227136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.227276 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.227701 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.227704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.227706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.228278 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.228538 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.228870 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.229060 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":542,"thread_start_us":341,"threads_started":1}
I20260927 22:54:21.231873 14044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:54:21 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 22:54:21 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:21.245738 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.232500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33137 (local address 127.0.0.1:35920)
0927 22:54:21.232646 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.232667 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:21.232795 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:21.233128 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:21.233132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:21.233509 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:21.233687 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.233699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.235067 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.235070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.236286 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.236300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.237212 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.237216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.237364 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.237955 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:21.237976 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:21.240065 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:21.243467 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:21.243473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:21.243477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:21.243837 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:21.244191 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:21.244197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:21.244201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:21.244344 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:21.244816 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:21.244820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:21.244965 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 22:54:21.245098 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 22:54:21.245370 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 22:54:21.245803 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.232586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.0.97.61:33137)
0927 22:54:21.232851 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.232856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.232873 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:54:21.232920 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.232924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.232987 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.233111 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.233842 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.234910 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.236418 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.237028 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.237173 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.240234 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.240239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.240244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.240287 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.243352 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.243955 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.243960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.243963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.244087 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.244444 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.244450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.244453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.244699 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.244852 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.245205 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.245420 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260927 22:54:21.246745 14044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:54:21.248654 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.233567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.0.97.62:43817)
0927 22:54:21.233752 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.233758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.233778 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:54:21.233899 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.233902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.233950 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.234016 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.235407 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.236451 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.237977 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.238632 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.238751 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.243104 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.243109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.243111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.243147 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.245749 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.246708 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.246712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.246714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.246829 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.247295 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.247300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.247302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.247632 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.247904 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.248152 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.248362 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 22:54:21 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:54:21 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:54:21.260354 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.247583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.0.97.60:36387)
0927 22:54:21.247741 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.247746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.247759 (+ 13us) server_negotiation.cc:407] Connection header received
0927 22:54:21.247801 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.247804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.247847 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.247955 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.248599 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.249431 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.251110 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.251498 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.251584 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.254730 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.254734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.254737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.254764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.257939 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.258610 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.258617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.258622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.258742 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.259149 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.259157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.259162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.259426 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.259571 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.259936 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.260082 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:54:21.260774 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.247402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39224)
0927 22:54:21.247534 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.247548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:21.247614 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:21.247968 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:21.247972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:21.248309 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:21.248430 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.248437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.249577 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.249582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.250966 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.250982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.251629 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.251634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.251853 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.252407 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:21.252422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:21.254565 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:21.258055 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:21.258058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:21.258061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:21.258504 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:21.258891 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:21.258894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:21.258896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:21.259049 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:21.259537 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:21.259542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:21.259697 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 22:54:21.259867 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 22:54:21.260331 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":96}
I20260927 22:54:21.265267 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.250112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.0.97.61:33137)
0927 22:54:21.250259 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.250262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.250276 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:54:21.250400 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.250403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.250447 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.250517 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.251989 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.252835 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.254366 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.254924 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.255015 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.259758 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.259764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.259768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.259803 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.263002 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.263557 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.263563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.263566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.263716 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.264041 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.264045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.264048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.264365 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.264625 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.264740 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.264955 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:54:21.266427 14044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4: Bootstrap starting.
I20260927 22:54:21.269728 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:54:21.269904 14044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:21.271102 14044 log.cc:824] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:21.274153 14044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4: No bootstrap required, opened a new log
I20260927 22:54:21.285705 14044 raft_consensus.cc:348] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } }
I20260927 22:54:21.286145 14044 raft_consensus.cc:374] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:21.287046 14044 raft_consensus.cc:729] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a67f92ebfbd74774b28a0f0610283bc4, State: Initialized, Role: FOLLOWER
I20260927 22:54:21.287938 14044 consensus_queue.cc:260] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } }
I20260927 22:54:21.294824 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.275460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.0.97.62:43817)
0927 22:54:21.275632 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.275638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.276330 (+ 692us) server_negotiation.cc:407] Connection header received
0927 22:54:21.276579 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.276584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.276655 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.276721 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.278997 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.280095 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.282320 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.282961 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.283063 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.286938 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.286946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.286950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.286994 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.291033 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.292245 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.292254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.292260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.292475 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.293060 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.293067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.293073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.293528 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.293847 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.294228 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.294446 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:54:21.296802 14044 sys_catalog.cc:564] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:21.298254 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } } }
I20260927 22:54:21.298710 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:54:21.304634 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.275269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.0.97.61:33137)
0927 22:54:21.275436 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.275442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.286820 (+ 11378us) server_negotiation.cc:407] Connection header received
0927 22:54:21.286998 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.287006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.287103 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.287223 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.289136 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.290322 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.292339 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.292978 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.293069 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.296606 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.296612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.296617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.296652 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.301024 (+ 4372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.301871 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.301877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.301880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.302033 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.302579 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.302584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.302588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.302968 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.303951 (+ 983us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.304108 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.304307 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:54:21.305087 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.277016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.0.97.60:36387)
0927 22:54:21.277239 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.277247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.288480 (+ 11233us) server_negotiation.cc:407] Connection header received
0927 22:54:21.288856 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.288863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.288937 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.289004 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.290626 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.291734 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.294099 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.294780 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.294892 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.298562 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.298568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.298573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.298608 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.302041 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.302902 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.302907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.302910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.303068 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.303523 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.303528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.303534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.303899 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.304141 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.304559 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.304795 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:54:21.310717 14068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:54:21.310905 14068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:54:21.308035 13977 master_runner.cc:186] Error getting master registration for 127.0.97.62:43817: OK, instance_id { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" instance_seqno: 1790549660209627 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:54:21.315790 13977 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:54:21.343737 13896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } attempt: 4
Sep 27 22:54:21 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:54:21.359230 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.344286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39236)
0927 22:54:21.344612 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.344630 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:21.344730 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:21.345399 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:21.345404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:21.345845 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:21.346017 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.346026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.347488 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.347492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.348391 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.348402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.349333 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.349337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.349465 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.350122 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:21.350139 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:21.352182 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:21.356577 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:21.356583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:21.356587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:21.357111 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:21.357642 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:21.357647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:21.357650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:21.357906 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:21.358504 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:21.358507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:21.358630 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 22:54:21.358736 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:54:21.358961 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260927 22:54:21.359478 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.344413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.0.97.60:36387)
0927 22:54:21.345080 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.345086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.345104 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:54:21.345158 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.345168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.345239 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.345356 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.346153 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.347352 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.348527 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.349201 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.349327 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.352378 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.352385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.352390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.352432 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.356372 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.357276 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.357281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.357285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.357479 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.358017 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.358023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.358028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.358386 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.358535 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.358885 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.359116 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538}
I20260927 22:54:21.363983 13896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216: Bootstrap starting.
I20260927 22:54:21.367435 13896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:21.368407 13896 log.cc:824] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:21.371003 13896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216: No bootstrap required, opened a new log
I20260927 22:54:21.378075 13896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } }
I20260927 22:54:21.378465 13896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:21.378612 13896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ca7ec6e03f42bbb0d8f62af0efa216, State: Initialized, Role: FOLLOWER
I20260927 22:54:21.379220 13896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } }
I20260927 22:54:21.380090 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } } }
I20260927 22:54:21.380455 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:54:21.381187 13896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:54:21.390792 14082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:54:21.390978 14082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:54:21.390092 13829 master_runner.cc:186] Error getting master registration for 127.0.97.61:33137: OK, instance_id { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" instance_seqno: 1790549660685900 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:54:21.398519 13829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:54:21.403868 14057 raft_consensus.cc:482] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:54:21.404217 14057 raft_consensus.cc:504] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } }
I20260927 22:54:21.405828 14057 leader_election.cc:288] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ca7ec6e03f42bbb0d8f62af0efa216 (127.0.97.62:43817), 4c2467608db74d41ba77e5f6da5c6dba (127.0.97.61:33137)
I20260927 22:54:21.406841 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a67f92ebfbd74774b28a0f0610283bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" is_pre_election: true
I20260927 22:54:21.407421 13871 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a67f92ebfbd74774b28a0f0610283bc4 in term 0.
I20260927 22:54:21.407990 13995 leader_election.cc:302] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ca7ec6e03f42bbb0d8f62af0efa216, a67f92ebfbd74774b28a0f0610283bc4; no voters:
I20260927 22:54:21.408381 14057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:54:21.408521 14057 raft_consensus.cc:482] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:54:21.408645 14057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:21.409055 13970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } attempt: 5
I20260927 22:54:21.412022 14057 raft_consensus.cc:504] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } }
I20260927 22:54:21.413053 14057 leader_election.cc:288] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [CANDIDATE]: Term 1 election: Requested vote from peers 44ca7ec6e03f42bbb0d8f62af0efa216 (127.0.97.62:43817), 4c2467608db74d41ba77e5f6da5c6dba (127.0.97.61:33137)
I20260927 22:54:21.413615 13871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a67f92ebfbd74774b28a0f0610283bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216"
I20260927 22:54:21.413823 13871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:21.417100 13871 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a67f92ebfbd74774b28a0f0610283bc4 in term 1.
I20260927 22:54:21.417512 13995 leader_election.cc:302] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ca7ec6e03f42bbb0d8f62af0efa216, a67f92ebfbd74774b28a0f0610283bc4; no voters:
I20260927 22:54:21.417963 14057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 22:54:21 dist-test-slave-df53 krb5kdc[13816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:54:21.429877 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.409529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36387 (local address 127.0.0.1:39240)
0927 22:54:21.409830 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.409852 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:21.409954 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:21.410828 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:21.410834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:21.411340 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:21.411533 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.411544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.413528 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.413533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.415039 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.415054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.416274 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.416280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.416463 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.417161 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:21.417187 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:21.420723 (+ 3536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:21.426496 (+ 5773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:21.426504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:21.426511 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:21.427147 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:21.427622 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:21.427629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:21.427634 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:21.427844 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:21.428676 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:21.428684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:21.428884 (+ 200us) client_negotiation.cc:769] Sending connection context
0927 22:54:21.429039 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 22:54:21.429370 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260927 22:54:21.431227 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.410147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39240 (local address 127.0.97.60:36387)
0927 22:54:21.410313 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.410324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.410341 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:54:21.410392 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.410397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.410465 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.410561 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.411707 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.412852 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.415258 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.415938 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.416067 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.420985 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.420994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.421002 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:21.421047 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.426290 (+ 5243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.427300 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.427309 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.427315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.427480 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:21.427982 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:21.427989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:21.427996 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:21.428432 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.428608 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.430414 (+ 1806us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.430762 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 22:54:21.432819 13998 tablet.cc:2369] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:54:21.438478 13970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba: Bootstrap starting.
I20260927 22:54:21.443328 13970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:21.445021 13970 log.cc:824] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:21.447732 14057 raft_consensus.cc:686] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [term 1 LEADER]: Becoming Leader. State: Replica: a67f92ebfbd74774b28a0f0610283bc4, State: Running, Role: LEADER
W20260927 22:54:21.448668 13850 tablet.cc:2369] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:54:21.448947 14057 consensus_queue.cc:237] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } }
I20260927 22:54:21.452987 13970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba: No bootstrap required, opened a new log
I20260927 22:54:21.462905 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a67f92ebfbd74774b28a0f0610283bc4. Latest consensus state: current_term: 1 leader_uuid: "a67f92ebfbd74774b28a0f0610283bc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } } }
I20260927 22:54:21.463421 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:21.467644 13970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } }
I20260927 22:54:21.468043 13970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:21.468175 13970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2467608db74d41ba77e5f6da5c6dba, State: Initialized, Role: FOLLOWER
I20260927 22:54:21.468827 13970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } }
I20260927 22:54:21.471307 13970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:21.471342 13944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a67f92ebfbd74774b28a0f0610283bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2467608db74d41ba77e5f6da5c6dba"
I20260927 22:54:21.471916 13944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:21.472622 13945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a67f92ebfbd74774b28a0f0610283bc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c2467608db74d41ba77e5f6da5c6dba" is_pre_election: true
I20260927 22:54:21.477106 13944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a67f92ebfbd74774b28a0f0610283bc4 in term 1.
I20260927 22:54:21.471060 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } } }
I20260927 22:54:21.477980 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:54:21.482115 14087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:21.484622 14087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:21.503495 13944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [term 1 FOLLOWER]: Refusing update from remote peer a67f92ebfbd74774b28a0f0610283bc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:21.504292 14084 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:21.511540 13871 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [term 1 FOLLOWER]: Refusing update from remote peer a67f92ebfbd74774b28a0f0610283bc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:21.512356 14084 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:21.543071 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a67f92ebfbd74774b28a0f0610283bc4. Latest consensus state: current_term: 1 leader_uuid: "a67f92ebfbd74774b28a0f0610283bc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } } }
I20260927 22:54:21.543421 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:54:21.545912 14090 mvcc.cc:205] Tried to move back new op lower bound from 7334091413509341184 to 7334091413305864192. Current Snapshot: MvccSnapshot[applied={T|T < 7334091413509341184}]
I20260927 22:54:21.546900 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a67f92ebfbd74774b28a0f0610283bc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } } }
I20260927 22:54:21.547142 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:21.550812 14087 catalog_manager.cc:1347] Generated new cluster ID: 95b40277d95440ca9fe946fd2d28c156
I20260927 22:54:21.550928 14087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:21.551154 14094 mvcc.cc:205] Tried to move back new op lower bound from 7334091413509341184 to 7334091413305864192. Current Snapshot: MvccSnapshot[applied={T|T < 7334091413509341184}]
I20260927 22:54:21.559036 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a67f92ebfbd74774b28a0f0610283bc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } } }
I20260927 22:54:21.559273 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:54:21.563303 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader a67f92ebfbd74774b28a0f0610283bc4. Latest consensus state: current_term: 1 leader_uuid: "a67f92ebfbd74774b28a0f0610283bc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } } }
I20260927 22:54:21.563622 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:54:21.564735 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a67f92ebfbd74774b28a0f0610283bc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } } }
I20260927 22:54:21.564965 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:21.572043 14092 mvcc.cc:205] Tried to move back new op lower bound from 7334091413509341184 to 7334091413305864192. Current Snapshot: MvccSnapshot[applied={T|T < 7334091413509341184}]
I20260927 22:54:21.586103 14087 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:54:21.587911 14087 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:21.588373 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a67f92ebfbd74774b28a0f0610283bc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ca7ec6e03f42bbb0d8f62af0efa216" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43817 } } peers { permanent_uuid: "4c2467608db74d41ba77e5f6da5c6dba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33137 } } peers { permanent_uuid: "a67f92ebfbd74774b28a0f0610283bc4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36387 } } }
I20260927 22:54:21.588616 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:54:21.601547 14087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a67f92ebfbd74774b28a0f0610283bc4: Generated new TSK 0
I20260927 22:54:21.602602 14087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:54:21.630656 14108 catalog_manager.cc:1259] Loaded cluster ID: 95b40277d95440ca9fe946fd2d28c156
I20260927 22:54:21.630851 14108 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba: loading cluster ID for follower catalog manager: success
I20260927 22:54:21.634534 14108 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba: acquiring CA information for follower catalog manager: success
I20260927 22:54:21.637141 14108 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c2467608db74d41ba77e5f6da5c6dba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:54:22.392951 14082 catalog_manager.cc:1259] Loaded cluster ID: 95b40277d95440ca9fe946fd2d28c156
I20260927 22:54:22.393054 14082 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216: loading cluster ID for follower catalog manager: success
I20260927 22:54:22.395433 14082 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216: acquiring CA information for follower catalog manager: success
I20260927 22:54:22.397219 14082 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44ca7ec6e03f42bbb0d8f62af0efa216: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:54:22.738209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 13829
I20260927 22:54:22.739547 13891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:22.847285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 13829
I20260927 22:54:22.867340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 13903
I20260927 22:54:22.868412 13965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:22.973515 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 13903
I20260927 22:54:22.993417 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 13977
I20260927 22:54:22.994562 14039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:23.104206 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 13977
2026-09-27T22:54:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4655 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:54:23 dist-test-slave-df53 krb5kdc[14121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:54:23 dist-test-slave-df53 krb5kdc[14121](info): set up 2 sockets
Sep 27 22:54:23 dist-test-slave-df53 krb5kdc[14121](info): commencing operation
krb5kdc: starting...
W20260927 22:54:24.210119 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:54:24 dist-test-slave-df53 krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:54:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:54:24.414906 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35587
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:24.727595 14134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:24.727916 14134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:24.728067 14134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:24.735169 14134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:24.735301 14134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:24.735400 14134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:24.735502 14134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:24.735574 14134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:24.745229 14134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:24.746527 14134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:24.747783 14134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:24.758903 14139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:24.758937 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:24.758960 14142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:24.759678 14134 server_base.cc:1034] running on GCE node
I20260927 22:54:24.760412 14134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:24.761804 14134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:24.763027 14134 hybrid_clock.cc:648] HybridClock initialized: now 1790549664762986 us; error 51 us; skew 500 ppm
I20260927 22:54:24.763357 14134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:24 dist-test-slave-df53 krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:24.766358 14134 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:54:24.767752 14134 webserver.cc:460] Webserver started at http://127.0.97.62:43013/ using document root <none> and password file <none>
I20260927 22:54:24.768363 14134 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:24.768468 14134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:24.768713 14134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:24.771559 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "770e37988f3c481c9cffd2492c1b60a1"
format_stamp: "Formatted at 2026-09-27 22:54:24 on dist-test-slave-df53"
server_key: "455fb1bc7b2bfa94e98ceecd9b5fc17c"
server_key_iv: "dec1820e90a8bbd5f3d248e6133f7dbf"
server_key_version: "encryptionkey@0"
I20260927 22:54:24.772226 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "770e37988f3c481c9cffd2492c1b60a1"
format_stamp: "Formatted at 2026-09-27 22:54:24 on dist-test-slave-df53"
server_key: "455fb1bc7b2bfa94e98ceecd9b5fc17c"
server_key_iv: "dec1820e90a8bbd5f3d248e6133f7dbf"
server_key_version: "encryptionkey@0"
I20260927 22:54:24.777045 14134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:54:24.780241 14149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:24.781482 14134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 22:54:24.781703 14134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "770e37988f3c481c9cffd2492c1b60a1"
format_stamp: "Formatted at 2026-09-27 22:54:24 on dist-test-slave-df53"
server_key: "455fb1bc7b2bfa94e98ceecd9b5fc17c"
server_key_iv: "dec1820e90a8bbd5f3d248e6133f7dbf"
server_key_version: "encryptionkey@0"
I20260927 22:54:24.781891 14134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:24.795315 14134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:24.799506 14134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:24.799785 14134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:24.816946 14134 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35587
I20260927 22:54:24.816967 14200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35587 every 8 connection(s)
I20260927 22:54:24.818276 14134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:24.821866 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:24.823609 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 14134
I20260927 22:54:24.823846 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:54:24.824651 388 external_mini_cluster.cc:1428] Setting key 6f759b965101d0bec3a6c4e7b175eb56
I20260927 22:54:24.836019 14201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1: Bootstrap starting.
Sep 27 22:54:24 dist-test-slave-df53 krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:24.839632 14201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:24.840682 14201 log.cc:824] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:24.843483 14201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1: No bootstrap required, opened a new log
I20260927 22:54:24.847304 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:24.826449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.0.97.62:35587)
0927 22:54:24.827235 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:24.827246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:24.827287 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:54:24.828843 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:24.828863 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:24.829222 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:24.829521 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:24.831313 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:24.832571 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:24.834466 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:24.835084 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:24.835227 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:24.840082 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:24.840089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:24.840102 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:24.840143 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:24.843209 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:24.844000 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:24.844006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:24.844012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:24.844180 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:24.844584 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:24.844588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:24.844590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:24.845147 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:24.845476 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:24.845960 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 22:54:24.846351 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":380,"threads_started":1}
I20260927 22:54:24.851164 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:54:24.852073 14201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770e37988f3c481c9cffd2492c1b60a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35587 } }
I20260927 22:54:24.852468 14201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:24.852574 14201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770e37988f3c481c9cffd2492c1b60a1, State: Initialized, Role: FOLLOWER
I20260927 22:54:24.853085 14201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770e37988f3c481c9cffd2492c1b60a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35587 } }
I20260927 22:54:24.853369 14201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:24.853513 14201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:24.853737 14201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:24.855950 14201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770e37988f3c481c9cffd2492c1b60a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35587 } }
I20260927 22:54:24.856359 14201 leader_election.cc:302] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 770e37988f3c481c9cffd2492c1b60a1; no voters:
I20260927 22:54:24.856812 14201 leader_election.cc:288] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:54:24.856956 14206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:24.857635 14206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [term 1 LEADER]: Becoming Leader. State: Replica: 770e37988f3c481c9cffd2492c1b60a1, State: Running, Role: LEADER
I20260927 22:54:24.858150 14206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770e37988f3c481c9cffd2492c1b60a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35587 } }
I20260927 22:54:24.858733 14201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:24.861034 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "770e37988f3c481c9cffd2492c1b60a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770e37988f3c481c9cffd2492c1b60a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35587 } } }
I20260927 22:54:24.861011 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770e37988f3c481c9cffd2492c1b60a1. Latest consensus state: current_term: 1 leader_uuid: "770e37988f3c481c9cffd2492c1b60a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770e37988f3c481c9cffd2492c1b60a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35587 } } }
I20260927 22:54:24.861454 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:24.861454 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:24.863878 14213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:24.868765 14213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:24.876880 14213 catalog_manager.cc:1347] Generated new cluster ID: 69ae2d3858f94a3792bbafd97e1a11f6
I20260927 22:54:24.876979 14213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:24.887341 14213 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:54:24.888520 14213 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:24.898274 14213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 770e37988f3c481c9cffd2492c1b60a1: Generated new TSK 0
I20260927 22:54:24.898964 14213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:54:24.911415 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 14134
I20260927 22:54:24.912461 14196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:25.002323 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 14134
2026-09-27T22:54:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1886 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:54:25 dist-test-slave-df53 krb5kdc[14229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:54:25 dist-test-slave-df53 krb5kdc[14229](info): set up 2 sockets
Sep 27 22:54:25 dist-test-slave-df53 krb5kdc[14229](info): commencing operation
krb5kdc: starting...
W20260927 22:54:26.094034 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:54:26 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:54:26.307291 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35627
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:26.624953 14242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:26.625247 14242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:26.625411 14242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:26.632354 14242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:26.632444 14242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:26.632573 14242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:26.632673 14242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:26.632756 14242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:26.642107 14242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:26.643363 14242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:26.644637 14242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:26.656124 14247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:26.656373 14250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:26.656296 14248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:26.657491 14242 server_base.cc:1034] running on GCE node
I20260927 22:54:26.658246 14242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:26.659549 14242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:26.660745 14242 hybrid_clock.cc:648] HybridClock initialized: now 1790549666660682 us; error 52 us; skew 500 ppm
I20260927 22:54:26.661041 14242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:26 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:26.670017 14242 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:54:26.671651 14242 webserver.cc:460] Webserver started at http://127.0.97.62:34631/ using document root <none> and password file <none>
I20260927 22:54:26.672350 14242 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:26.672490 14242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:26.672814 14242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:26.675882 14242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "0fec4059bbaf4faf9c395d38758ffeeb"
format_stamp: "Formatted at 2026-09-27 22:54:26 on dist-test-slave-df53"
server_key: "da5a6c328a6af2bcd9063be3fe118a69"
server_key_iv: "115d1685336cfe5389e64354d0accd80"
server_key_version: "encryptionkey@0"
I20260927 22:54:26.676659 14242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "0fec4059bbaf4faf9c395d38758ffeeb"
format_stamp: "Formatted at 2026-09-27 22:54:26 on dist-test-slave-df53"
server_key: "da5a6c328a6af2bcd9063be3fe118a69"
server_key_iv: "115d1685336cfe5389e64354d0accd80"
server_key_version: "encryptionkey@0"
I20260927 22:54:26.681828 14242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:54:26.685191 14257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:26.686556 14242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:26.686717 14242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "0fec4059bbaf4faf9c395d38758ffeeb"
format_stamp: "Formatted at 2026-09-27 22:54:26 on dist-test-slave-df53"
server_key: "da5a6c328a6af2bcd9063be3fe118a69"
server_key_iv: "115d1685336cfe5389e64354d0accd80"
server_key_version: "encryptionkey@0"
I20260927 22:54:26.686918 14242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:26.704278 14242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:26.708698 14242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:26.709014 14242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:26.727461 14242 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35627
I20260927 22:54:26.727495 14308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35627 every 8 connection(s)
I20260927 22:54:26.728806 14242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:26.732496 14309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:26.737177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 14242
I20260927 22:54:26.737402 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:54:26.738296 388 external_mini_cluster.cc:1428] Setting key f0704618a040d896f32c11c9d43ba043
I20260927 22:54:26.744107 14309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Bootstrap starting.
I20260927 22:54:26.747705 14309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:26.748915 14309 log.cc:824] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:26.752485 14309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: No bootstrap required, opened a new log
Sep 27 22:54:26 dist-test-slave-df53 krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:26.760210 14309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:26.760627 14309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:26.760748 14309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fec4059bbaf4faf9c395d38758ffeeb, State: Initialized, Role: FOLLOWER
I20260927 22:54:26.761410 14309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:26.761713 14309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:26.761868 14309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:26.762050 14309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:26.762336 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:26.740312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.0.97.62:35627)
0927 22:54:26.741128 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:26.741139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:26.741179 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:54:26.742932 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:26.742954 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:26.743322 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:26.743650 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:26.745381 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:26.746746 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:26.748683 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:26.749573 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:26.749859 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:26.754537 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:26.754545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:26.754560 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:26.754601 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:26.757882 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:26.758600 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:26.758610 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:26.758618 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:26.758798 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:26.759208 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:26.759211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:26.759214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:26.759751 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:26.760062 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:26.760500 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 22:54:26.761124 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":385,"threads_started":1}
I20260927 22:54:26.764446 14309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:26.764941 14309 leader_election.cc:302] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fec4059bbaf4faf9c395d38758ffeeb; no voters:
I20260927 22:54:26.765522 14309 leader_election.cc:288] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:54:26.765689 14314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:26.766455 14314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 1 LEADER]: Becoming Leader. State: Replica: 0fec4059bbaf4faf9c395d38758ffeeb, State: Running, Role: LEADER
I20260927 22:54:26.766979 14314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:26.767637 14309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:26.769867 14315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } } }
I20260927 22:54:26.769910 14316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fec4059bbaf4faf9c395d38758ffeeb. Latest consensus state: current_term: 1 leader_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } } }
I20260927 22:54:26.770241 14315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:26.770241 14316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:26.773950 14321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:26.779110 14321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:26.789043 14321 catalog_manager.cc:1347] Generated new cluster ID: f207467b3e3b44e99de108cf392a224e
I20260927 22:54:26.789158 14321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:26.815164 14321 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:54:26.816515 14321 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:26.830642 14321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Generated new TSK 0
I20260927 22:54:26.831534 14321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:54:26.883039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35627
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:27.204584 14335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:27.204845 14335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:27.204954 14335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:27.212261 14335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:27.212384 14335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:27.212548 14335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:54:27.222011 14335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:27.223174 14335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:27.224388 14335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:27.236855 14340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:27.236857 14341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:27.237318 14343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:27.237624 14335 server_base.cc:1034] running on GCE node
I20260927 22:54:27.238214 14335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:27.239116 14335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:27.240351 14335 hybrid_clock.cc:648] HybridClock initialized: now 1790549667240283 us; error 51 us; skew 500 ppm
I20260927 22:54:27.240649 14335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:27 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:27.243685 14335 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:54:27.245165 14335 webserver.cc:460] Webserver started at http://127.0.97.1:38399/ using document root <none> and password file <none>
I20260927 22:54:27.245885 14335 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:27.246024 14335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:27.246454 14335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:27.249325 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "3b509e932bed4262bc10cf887cf1f7f6"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-df53"
server_key: "808886ef724c1eae5baaca1d39cad6d3"
server_key_iv: "392d24168662b0d84466eb5d9ea3c8a4"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.250138 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3b509e932bed4262bc10cf887cf1f7f6"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-df53"
server_key: "808886ef724c1eae5baaca1d39cad6d3"
server_key_iv: "392d24168662b0d84466eb5d9ea3c8a4"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.255025 14335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:54:27.258455 14350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.259738 14335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:54:27.259934 14335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "3b509e932bed4262bc10cf887cf1f7f6"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-df53"
server_key: "808886ef724c1eae5baaca1d39cad6d3"
server_key_iv: "392d24168662b0d84466eb5d9ea3c8a4"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.260136 14335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:27.278337 14335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:27.282152 14335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:27.282437 14335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:27.283396 14335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:27.284684 14335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:27.284793 14335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.284950 14335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:27.285022 14335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.318372 14335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42955
I20260927 22:54:27.318419 14462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42955 every 8 connection(s)
I20260927 22:54:27.319608 14335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:27.324187 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 14335
I20260927 22:54:27.324429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:54:27.325218 388 external_mini_cluster.cc:1428] Setting key aaa2acc5586634847180e03713e0fcf9
Sep 27 22:54:27 dist-test-slave-df53 krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:27.341526 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:27.322113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51941 (local address 127.0.97.62:35627)
0927 22:54:27.322542 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:27.322547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:27.323499 (+ 952us) server_negotiation.cc:407] Connection header received
0927 22:54:27.326100 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:27.326106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:27.326195 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:27.326311 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:27.328017 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.328731 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.331244 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.331632 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.331730 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:27.335602 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:27.335607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:27.335610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:27.335671 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:27.338344 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:27.339217 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:27.339221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:27.339223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:27.339382 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:27.339848 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:27.339851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:27.339853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:27.340145 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:27.340255 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:27.341099 (+ 844us) server_negotiation.cc:299] Negotiation successful
0927 22:54:27.341258 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":228,"threads_started":1}
I20260927 22:54:27.343009 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:27.322699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:51941)
0927 22:54:27.323344 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:27.323389 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:27.325821 (+ 2432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:27.326546 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:27.326555 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:27.327076 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:27.327823 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:27.327859 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.328898 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.328905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:27.331104 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:27.331121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.331867 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.331877 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:27.332205 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:27.332990 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:27.333016 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:27.335381 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:27.338474 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:27.338485 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:27.338503 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:27.339101 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:27.339489 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:27.339496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:27.339503 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:27.339738 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:27.340276 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:27.340286 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:27.340728 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 22:54:27.341009 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 22:54:27.341550 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":257,"threads_started":1}
I20260927 22:54:27.344528 14463 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35627
I20260927 22:54:27.344848 14463 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:27.345551 14463 heartbeater.cc:511] Master 127.0.97.62:35627 requested a full tablet report, sending...
I20260927 22:54:27.347565 14274 ts_manager.cc:194] Registered new tserver with Master: 3b509e932bed4262bc10cf887cf1f7f6 (127.0.97.1:42955)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:54:27.349052 14274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:54:27.376593 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35627
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:27.693111 14469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:27.693377 14469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:27.693529 14469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:27.700784 14469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:27.700887 14469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:27.701071 14469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:54:27.710484 14469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:27.711781 14469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:27.713023 14469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:27.725771 14474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:27.725819 14475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:27.725857 14477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:27.726611 14469 server_base.cc:1034] running on GCE node
I20260927 22:54:27.727283 14469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:27.728163 14469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:27.729393 14469 hybrid_clock.cc:648] HybridClock initialized: now 1790549667729340 us; error 46 us; skew 500 ppm
I20260927 22:54:27.729766 14469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:27 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:27.732933 14469 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:54:27.734392 14469 webserver.cc:460] Webserver started at http://127.0.97.2:36257/ using document root <none> and password file <none>
I20260927 22:54:27.735019 14469 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:27.735148 14469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:27.735448 14469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:27.738227 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "e2a47bc0e167486f9ef0ce77abb268ea"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-df53"
server_key: "a46247804e8eff0de0d83c13541850ed"
server_key_iv: "0718bf04f9cb24bfd08a791e43171327"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.739019 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e2a47bc0e167486f9ef0ce77abb268ea"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-df53"
server_key: "a46247804e8eff0de0d83c13541850ed"
server_key_iv: "0718bf04f9cb24bfd08a791e43171327"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.743932 14469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:54:27.747340 14484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.748522 14469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:54:27.748682 14469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "e2a47bc0e167486f9ef0ce77abb268ea"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-df53"
server_key: "a46247804e8eff0de0d83c13541850ed"
server_key_iv: "0718bf04f9cb24bfd08a791e43171327"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.748960 14469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:27.777112 14469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:27.781162 14469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:27.781451 14469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:27.782485 14469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:27.783771 14469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:27.783843 14469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.784013 14469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:27.784063 14469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.816939 14469 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39339
I20260927 22:54:27.817022 14596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39339 every 8 connection(s)
I20260927 22:54:27.818176 14469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:27.821884 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 14469
I20260927 22:54:27.822069 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:54:27.822651 388 external_mini_cluster.cc:1428] Setting key 8e486daa64a4d527caf216397e327ac7
Sep 27 22:54:27 dist-test-slave-df53 krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:27.839125 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:27.820676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57777 (local address 127.0.97.62:35627)
0927 22:54:27.820862 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:27.820867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:27.822207 (+ 1340us) server_negotiation.cc:407] Connection header received
0927 22:54:27.824703 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:27.824709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:27.824791 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:27.824932 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:27.826437 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.827149 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.828773 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.829173 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.829313 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:27.833456 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:27.833461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:27.833464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:27.833495 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:27.836098 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:27.836842 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:27.836846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:27.836848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:27.836970 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:27.837387 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:27.837399 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:27.837402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:27.837732 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:27.837897 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:27.838557 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 22:54:27.838791 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:54:27.840070 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:27.821141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:57777)
0927 22:54:27.822024 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:27.822080 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:27.824414 (+ 2334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:27.825158 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:27.825166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:27.825724 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:27.826271 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:27.826290 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.827299 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.827304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:27.828616 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:27.828627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.829397 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.829405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:27.829686 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:27.830353 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:27.830374 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:27.833229 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:27.836240 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:27.836250 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:27.836265 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:27.836699 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:27.837080 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:27.837086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:27.837090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:27.837262 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:27.837864 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:27.837872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:27.838189 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 22:54:27.838454 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 22:54:27.838935 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":324,"threads_started":1}
I20260927 22:54:27.841554 14597 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35627
I20260927 22:54:27.841941 14597 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:27.842603 14597 heartbeater.cc:511] Master 127.0.97.62:35627 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:54:27.844105 14274 ts_manager.cc:194] Registered new tserver with Master: e2a47bc0e167486f9ef0ce77abb268ea (127.0.97.2:39339)
I20260927 22:54:27.844930 14274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:54:27.872728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35627
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:28.184368 14602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:28.184623 14602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:28.184715 14602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:28.191826 14602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:28.191923 14602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:28.192037 14602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:54:28.201896 14602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:28.202989 14602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:28.204155 14602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:28.217549 14608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:28.217583 14610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:28.217662 14607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:28.218086 14602 server_base.cc:1034] running on GCE node
I20260927 22:54:28.218875 14602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:28.219673 14602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:28.220968 14602 hybrid_clock.cc:648] HybridClock initialized: now 1790549668220916 us; error 58 us; skew 500 ppm
I20260927 22:54:28.221307 14602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:28 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:28.224334 14602 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:54:28.225697 14602 webserver.cc:460] Webserver started at http://127.0.97.3:34133/ using document root <none> and password file <none>
I20260927 22:54:28.226358 14602 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:28.226500 14602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:28.226786 14602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:28.229590 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "e18a873f084549f9ad7b5c384818ba15"
format_stamp: "Formatted at 2026-09-27 22:54:28 on dist-test-slave-df53"
server_key: "d9b3f9e7e58a4906292f48255e9cfd15"
server_key_iv: "a87608d04cb9c8473e766b491257876b"
server_key_version: "encryptionkey@0"
I20260927 22:54:28.230432 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e18a873f084549f9ad7b5c384818ba15"
format_stamp: "Formatted at 2026-09-27 22:54:28 on dist-test-slave-df53"
server_key: "d9b3f9e7e58a4906292f48255e9cfd15"
server_key_iv: "a87608d04cb9c8473e766b491257876b"
server_key_version: "encryptionkey@0"
I20260927 22:54:28.235306 14602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:54:28.238608 14617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:28.239831 14602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:28.240027 14602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "e18a873f084549f9ad7b5c384818ba15"
format_stamp: "Formatted at 2026-09-27 22:54:28 on dist-test-slave-df53"
server_key: "d9b3f9e7e58a4906292f48255e9cfd15"
server_key_iv: "a87608d04cb9c8473e766b491257876b"
server_key_version: "encryptionkey@0"
I20260927 22:54:28.240211 14602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:28.284695 14602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:28.288594 14602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:28.288897 14602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:28.289898 14602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:28.291230 14602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:28.291337 14602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:28.291491 14602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:28.291563 14602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:28.324703 14602 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39533
I20260927 22:54:28.324725 14729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39533 every 8 connection(s)
I20260927 22:54:28.326131 14602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:28.333240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 14602
I20260927 22:54:28.333446 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:54:28.333954 388 external_mini_cluster.cc:1428] Setting key f399d3cdcfa0632c0305620f74b6d73f
Sep 27 22:54:28 dist-test-slave-df53 krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:28.346078 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.328533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42401 (local address 127.0.97.62:35627)
0927 22:54:28.328714 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.328719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.329852 (+ 1133us) server_negotiation.cc:407] Connection header received
0927 22:54:28.331656 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.331662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.331748 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.331860 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:28.333503 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.334454 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.336200 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.336847 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.336985 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.340384 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:28.340389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:28.340394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:28.340428 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:28.343264 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.344062 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.344066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.344068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.344188 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.344654 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.344657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.344659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.344926 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:28.345036 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.345659 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.345814 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:54:28.346988 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.328995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:42401)
0927 22:54:28.329708 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.329748 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.331395 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.332083 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.332090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.332568 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:28.333269 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.333294 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.334608 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.334612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.336071 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.336083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.337069 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.337077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.337324 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.338002 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:28.338023 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:28.340174 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:28.343457 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:28.343463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:28.343474 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:28.343935 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:28.344309 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:28.344312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:28.344315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:28.344542 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:28.345030 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:28.345035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:28.345308 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.345520 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.345956 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":264,"threads_started":1}
I20260927 22:54:28.348523 14730 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35627
I20260927 22:54:28.348908 14730 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:28.349714 14730 heartbeater.cc:511] Master 127.0.97.62:35627 requested a full tablet report, sending...
I20260927 22:54:28.350955 14274 ts_manager.cc:194] Registered new tserver with Master: e18a873f084549f9ad7b5c384818ba15 (127.0.97.3:39533)
I20260927 22:54:28.351686 14274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42401
I20260927 22:54:28.351948 14463 heartbeater.cc:503] Master 127.0.97.62:35627 was elected leader, sending a full tablet report...
I20260927 22:54:28.361268 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:54:28.378665 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.365791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.0.97.62:35627)
0927 22:54:28.365967 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.365990 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.366424 (+ 434us) server_negotiation.cc:407] Connection header received
0927 22:54:28.366625 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.366629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.366686 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.366788 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:28.368148 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.368849 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.370554 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.371189 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.371325 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.373629 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:28.373636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:28.373640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:28.373676 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:28.376049 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.376718 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.376725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.376728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.376852 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.377224 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.377230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.377233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.377543 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:28.377818 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.378132 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.378348 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:54:28.382409 14274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:54:28.388999 14274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:54:28.414353 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.401961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.0.97.3:39533)
0927 22:54:28.404598 (+ 2637us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.404610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.405301 (+ 691us) server_negotiation.cc:407] Connection header received
0927 22:54:28.405413 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.405421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.405672 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.405872 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:28.406687 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.408170 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.411783 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.413088 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.413241 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.413463 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.413791 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.413975 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2422,"thread_start_us":2231,"threads_started":1}
I20260927 22:54:28.414388 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.401273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39533 (local address 127.0.0.1:53526)
0927 22:54:28.405051 (+ 3778us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.405065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.405203 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.406021 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.406026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.406050 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:28.406477 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.406489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.408330 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.408335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.411576 (+ 3241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.411595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.413269 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.413275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.413480 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.413545 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.413715 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.413898 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3661,"mutex_wait_us":49,"thread_start_us":2028,"threads_started":1}
I20260927 22:54:28.414577 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.401346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.0.97.1:42955)
0927 22:54:28.402436 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.402449 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.404663 (+ 2214us) server_negotiation.cc:407] Connection header received
0927 22:54:28.404811 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.404818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.405083 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.406197 (+ 1114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:28.406470 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.408972 (+ 2502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.411765 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.413213 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.413340 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.413573 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.414079 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.414246 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":883,"threads_started":1}
I20260927 22:54:28.415515 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.401221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.0.1:52484)
0927 22:54:28.403765 (+ 2544us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.403826 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.404527 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.405364 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.405370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.405400 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:28.405858 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.405874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.409082 (+ 3208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.409089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.411589 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.411604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.413670 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.413680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.413860 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.413889 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.414985 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.415123 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2239,"mutex_wait_us":94,"thread_start_us":3571,"threads_started":1}
I20260927 22:54:28.418203 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.405163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39339 (local address 127.0.0.1:51394)
0927 22:54:28.409960 (+ 4797us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.409981 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.410113 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.410913 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.410918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.410943 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:28.411359 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.411369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.413405 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.413410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.415401 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.415413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.417412 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.417418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.417641 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.417674 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.417769 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.417893 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4663,"thread_start_us":4506,"threads_started":1}
I20260927 22:54:28.418296 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.405494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.0.97.2:39339)
0927 22:54:28.409480 (+ 3986us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.409489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.410090 (+ 601us) server_negotiation.cc:407] Connection header received
0927 22:54:28.410248 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.410256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.410485 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.410654 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:28.411733 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.413240 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.415554 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.416967 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.417096 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.417274 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.417823 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.417991 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3815,"thread_start_us":3675,"threads_started":1}
I20260927 22:54:28.418568 14665 tablet_service.cc:1467] Processing CreateTablet for tablet 512db29e8d5b44caae7db9dc8b476aac (DEFAULT_TABLE table=test-table [id=de96083ddcf145edb088186d68495352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:28.419057 14398 tablet_service.cc:1467] Processing CreateTablet for tablet 512db29e8d5b44caae7db9dc8b476aac (DEFAULT_TABLE table=test-table [id=de96083ddcf145edb088186d68495352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:28.419766 14665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512db29e8d5b44caae7db9dc8b476aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:28.420044 14398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512db29e8d5b44caae7db9dc8b476aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:28.424419 14532 tablet_service.cc:1467] Processing CreateTablet for tablet 512db29e8d5b44caae7db9dc8b476aac (DEFAULT_TABLE table=test-table [id=de96083ddcf145edb088186d68495352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:28.425325 14532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512db29e8d5b44caae7db9dc8b476aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:28.432715 14748 tablet_bootstrap.cc:492] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: Bootstrap starting.
I20260927 22:54:28.436142 14749 tablet_bootstrap.cc:492] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Bootstrap starting.
I20260927 22:54:28.436391 14748 tablet_bootstrap.cc:654] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:28.438271 14750 tablet_bootstrap.cc:492] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: Bootstrap starting.
I20260927 22:54:28.439066 14748 log.cc:824] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:28.439332 14749 tablet_bootstrap.cc:654] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:28.440574 14749 log.cc:824] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:28.441675 14750 tablet_bootstrap.cc:654] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:28.442065 14748 tablet_bootstrap.cc:492] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: No bootstrap required, opened a new log
I20260927 22:54:28.442315 14748 ts_tablet_manager.cc:1397] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:54:28.442843 14750 log.cc:824] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:28.443687 14749 tablet_bootstrap.cc:492] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: No bootstrap required, opened a new log
I20260927 22:54:28.443988 14749 ts_tablet_manager.cc:1397] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:54:28.450470 14748 raft_consensus.cc:348] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } }
I20260927 22:54:28.450856 14748 raft_consensus.cc:374] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:28.450980 14748 raft_consensus.cc:729] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18a873f084549f9ad7b5c384818ba15, State: Initialized, Role: FOLLOWER
I20260927 22:54:28.451635 14748 consensus_queue.cc:260] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } }
I20260927 22:54:28.452184 14750 tablet_bootstrap.cc:492] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: No bootstrap required, opened a new log
I20260927 22:54:28.452485 14750 ts_tablet_manager.cc:1397] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.005s
I20260927 22:54:28.452687 14730 heartbeater.cc:503] Master 127.0.97.62:35627 was elected leader, sending a full tablet report...
I20260927 22:54:28.453375 14748 ts_tablet_manager.cc:1428] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260927 22:54:28.455802 14749 raft_consensus.cc:348] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } }
I20260927 22:54:28.456239 14749 raft_consensus.cc:374] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:28.456355 14749 raft_consensus.cc:729] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b509e932bed4262bc10cf887cf1f7f6, State: Initialized, Role: FOLLOWER
I20260927 22:54:28.456955 14749 consensus_queue.cc:260] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } }
I20260927 22:54:28.458627 14749 ts_tablet_manager.cc:1428] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 22:54:28.462314 14750 raft_consensus.cc:348] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } }
I20260927 22:54:28.462756 14750 raft_consensus.cc:374] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:28.462854 14750 raft_consensus.cc:729] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a47bc0e167486f9ef0ce77abb268ea, State: Initialized, Role: FOLLOWER
I20260927 22:54:28.463374 14750 consensus_queue.cc:260] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } }
I20260927 22:54:28.464378 14597 heartbeater.cc:503] Master 127.0.97.62:35627 was elected leader, sending a full tablet report...
I20260927 22:54:28.465001 14750 ts_tablet_manager.cc:1428] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
W20260927 22:54:28.570149 14598 tablet.cc:2369] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:54:28.571915 14464 tablet.cc:2369] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:54:28.577621 14731 tablet.cc:2369] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:54:28.830576 14754 raft_consensus.cc:482] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:54:28.830924 14754 raft_consensus.cc:504] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } }
I20260927 22:54:28.832427 14754 leader_election.cc:288] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b509e932bed4262bc10cf887cf1f7f6 (127.0.97.1:42955), e2a47bc0e167486f9ef0ce77abb268ea (127.0.97.2:39339)
I20260927 22:54:28.838083 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.832819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39261 (local address 127.0.97.1:42955)
0927 22:54:28.833029 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.833035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.833078 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:54:28.833267 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.833273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.833378 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.833484 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:28.834003 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.834798 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.836378 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.837155 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.837308 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.837409 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.837716 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.837836 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:54:28.838102 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.832841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.97.3:39261)
0927 22:54:28.833002 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.833021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.833130 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.833521 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.833525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.833556 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:28.833837 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.833858 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.834972 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.834977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.836211 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.836223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.837315 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.837321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.837507 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.837531 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.837690 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.837822 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":1}
I20260927 22:54:28.839007 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.833287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39339 (local address 127.0.97.3:52349)
0927 22:54:28.833990 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.834013 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.834135 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.834441 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.834445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.834470 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:28.834752 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.834760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.835760 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.835765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.837036 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.837047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.838212 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.838218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.838497 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.838537 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.838637 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.838755 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":455,"threads_started":1}
I20260927 22:54:28.839071 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.833531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52349 (local address 127.0.97.2:39339)
0927 22:54:28.833822 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.833828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.834085 (+ 257us) server_negotiation.cc:407] Connection header received
0927 22:54:28.834261 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.834266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.834330 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.834425 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:28.834914 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.835607 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.837185 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.838004 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.838148 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.838240 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.838672 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.838788 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260927 22:54:28.839298 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512db29e8d5b44caae7db9dc8b476aac" candidate_uuid: "e18a873f084549f9ad7b5c384818ba15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b509e932bed4262bc10cf887cf1f7f6" is_pre_election: true
I20260927 22:54:28.839862 14418 raft_consensus.cc:2454] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e18a873f084549f9ad7b5c384818ba15 in term 0.
I20260927 22:54:28.840013 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512db29e8d5b44caae7db9dc8b476aac" candidate_uuid: "e18a873f084549f9ad7b5c384818ba15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" is_pre_election: true
I20260927 22:54:28.840502 14552 raft_consensus.cc:2454] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e18a873f084549f9ad7b5c384818ba15 in term 0.
I20260927 22:54:28.840519 14619 leader_election.cc:302] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b509e932bed4262bc10cf887cf1f7f6, e18a873f084549f9ad7b5c384818ba15; no voters:
I20260927 22:54:28.840893 14754 raft_consensus.cc:2781] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:54:28.841046 14754 raft_consensus.cc:482] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:54:28.841179 14754 raft_consensus.cc:3037] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:28.843473 14754 raft_consensus.cc:504] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } }
I20260927 22:54:28.844008 14754 leader_election.cc:288] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [CANDIDATE]: Term 1 election: Requested vote from peers 3b509e932bed4262bc10cf887cf1f7f6 (127.0.97.1:42955), e2a47bc0e167486f9ef0ce77abb268ea (127.0.97.2:39339)
I20260927 22:54:28.844519 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512db29e8d5b44caae7db9dc8b476aac" candidate_uuid: "e18a873f084549f9ad7b5c384818ba15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b509e932bed4262bc10cf887cf1f7f6"
I20260927 22:54:28.844619 14552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512db29e8d5b44caae7db9dc8b476aac" candidate_uuid: "e18a873f084549f9ad7b5c384818ba15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a47bc0e167486f9ef0ce77abb268ea"
I20260927 22:54:28.844743 14418 raft_consensus.cc:3037] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:28.844810 14552 raft_consensus.cc:3037] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:28.846935 14552 raft_consensus.cc:2454] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e18a873f084549f9ad7b5c384818ba15 in term 1.
I20260927 22:54:28.847164 14418 raft_consensus.cc:2454] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e18a873f084549f9ad7b5c384818ba15 in term 1.
I20260927 22:54:28.847304 14619 leader_election.cc:302] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e18a873f084549f9ad7b5c384818ba15, e2a47bc0e167486f9ef0ce77abb268ea; no voters:
I20260927 22:54:28.847591 14754 raft_consensus.cc:2781] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:28.848043 14754 raft_consensus.cc:686] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 1 LEADER]: Becoming Leader. State: Replica: e18a873f084549f9ad7b5c384818ba15, State: Running, Role: LEADER
I20260927 22:54:28.848466 14754 consensus_queue.cc:237] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } }
I20260927 22:54:28.852898 14274 catalog_manager.cc:5495] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 reported cstate change: term changed from 0 to 1, leader changed from <none> to e18a873f084549f9ad7b5c384818ba15 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e18a873f084549f9ad7b5c384818ba15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } health_report { overall_health: UNKNOWN } } }
I20260927 22:54:28.973968 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.968087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.0.97.3:39533)
0927 22:54:28.968637 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.968644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.968770 (+ 126us) server_negotiation.cc:407] Connection header received
0927 22:54:28.969027 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.969032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.969109 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.969273 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:54:28.969737 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.970396 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.971890 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.972276 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.972415 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.972886 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:54:28.973366 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:54:28.973518 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.973585 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.973688 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":288,"threads_started":1}
I20260927 22:54:28.981354 14552 raft_consensus.cc:1261] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea [term 1 FOLLOWER]: Refusing update from remote peer e18a873f084549f9ad7b5c384818ba15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:28.981364 14418 raft_consensus.cc:1261] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6 [term 1 FOLLOWER]: Refusing update from remote peer e18a873f084549f9ad7b5c384818ba15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:28.982318 14758 consensus_queue.cc:1035] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:28.982668 14754 consensus_queue.cc:1035] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:28.992749 14763 mvcc.cc:205] Tried to move back new op lower bound from 7334091444138504192 to 7334091443607064576. Current Snapshot: MvccSnapshot[applied={T|T < 7334091444138504192}]
I20260927 22:54:29.006285 14274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:54:29.006559 14274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:54:29.009990 14274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Sending DeleteTablet for 3 replicas of tablet 512db29e8d5b44caae7db9dc8b476aac
I20260927 22:54:29.011119 14398 tablet_service.cc:1514] Processing DeleteTablet for tablet 512db29e8d5b44caae7db9dc8b476aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52484
I20260927 22:54:29.011119 14665 tablet_service.cc:1514] Processing DeleteTablet for tablet 512db29e8d5b44caae7db9dc8b476aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53526
I20260927 22:54:29.011307 14532 tablet_service.cc:1514] Processing DeleteTablet for tablet 512db29e8d5b44caae7db9dc8b476aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51394
I20260927 22:54:29.011659 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 14335
I20260927 22:54:29.026170 14777 tablet_replica.cc:330] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: stopping tablet replica
I20260927 22:54:29.026731 14777 raft_consensus.cc:2229] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:54:29.028628 14777 raft_consensus.cc:2258] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:29.029623 14776 tablet_replica.cc:330] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: stopping tablet replica
I20260927 22:54:29.032176 14777 ts_tablet_manager.cc:1905] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:29.032791 14776 raft_consensus.cc:2229] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:54:29.033385 14776 raft_consensus.cc:2258] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 22:54:29 dist-test-slave-df53 krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:54:29.036082 14776 ts_tablet_manager.cc:1905] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:29.038178 14775 tablet_replica.cc:330] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: stopping tablet replica
I20260927 22:54:29.038653 14775 raft_consensus.cc:2229] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:54:29.043756 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:29.012746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.0.97.1:42955)
0927 22:54:29.012933 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:29.012939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:29.022397 (+ 9458us) server_negotiation.cc:407] Connection header received
0927 22:54:29.023330 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:29.023336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:29.023459 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:29.023544 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:29.026443 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:29.027621 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:29.029674 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:29.030353 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:29.030448 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:29.036044 (+ 5596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:29.036058 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:29.036079 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:29.036117 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:29.040143 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:29.041011 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:29.041019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:29.041023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:29.041186 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:29.041661 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:29.041666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:29.041669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:29.042137 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:29.042457 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:29.043086 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 22:54:29.043333 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:54:29.044656 14775 raft_consensus.cc:2258] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:29.045152 14458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:29.046816 14777 ts_tablet_manager.cc:1918] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:29.046984 14777 log.cc:1196] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/512db29e8d5b44caae7db9dc8b476aac
I20260927 22:54:29.047005 14775 ts_tablet_manager.cc:1905] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:29.047498 14777 ts_tablet_manager.cc:1939] T 512db29e8d5b44caae7db9dc8b476aac P e2a47bc0e167486f9ef0ce77abb268ea: Deleting consensus metadata
I20260927 22:54:29.049348 14259 catalog_manager.cc:4841] TS e2a47bc0e167486f9ef0ce77abb268ea (127.0.97.2:39339): tablet 512db29e8d5b44caae7db9dc8b476aac (table test-table [id=de96083ddcf145edb088186d68495352]) successfully deleted
I20260927 22:54:29.055819 14775 ts_tablet_manager.cc:1918] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:29.055992 14775 log.cc:1196] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/512db29e8d5b44caae7db9dc8b476aac
I20260927 22:54:29.056543 14775 ts_tablet_manager.cc:1939] T 512db29e8d5b44caae7db9dc8b476aac P e18a873f084549f9ad7b5c384818ba15: Deleting consensus metadata
I20260927 22:54:29.058260 14258 catalog_manager.cc:4841] TS e18a873f084549f9ad7b5c384818ba15 (127.0.97.3:39533): tablet 512db29e8d5b44caae7db9dc8b476aac (table test-table [id=de96083ddcf145edb088186d68495352]) successfully deleted
I20260927 22:54:29.139057 14776 ts_tablet_manager.cc:1918] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:29.139313 14776 log.cc:1196] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/512db29e8d5b44caae7db9dc8b476aac
I20260927 22:54:29.139653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 14335
W20260927 22:54:29.157577 14259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42955 (error 108)
I20260927 22:54:29.157877 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 14469
I20260927 22:54:29.162554 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:29.159921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.0.1:52498)
0927 22:54:29.162219 (+ 2298us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:29.162326 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2184,"thread_start_us":2090,"threads_started":1}
W20260927 22:54:29.162828 14259 catalog_manager.cc:4584] TS 3b509e932bed4262bc10cf887cf1f7f6 (127.0.97.1:42955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 512db29e8d5b44caae7db9dc8b476aac: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42955: connect: Connection refused (error 111)
Sep 27 22:54:29 dist-test-slave-df53 krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:54:29.175913 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:29.158761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.0.97.2:39339)
0927 22:54:29.158992 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:29.159000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:29.159026 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:54:29.159208 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:29.159216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:29.159326 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:29.159431 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:29.161657 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:29.162797 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:29.164289 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:29.164943 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:29.165057 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:29.169433 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:29.169450 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:29.169471 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:29.169532 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:29.173127 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:29.173841 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:29.173851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:29.173856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:29.174006 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:29.174340 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:29.174345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:29.174348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:29.174826 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:29.175118 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:29.175326 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 22:54:29.175542 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:54:29.176623 14592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:29.216703 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:29.216178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.0.1:52504)
0927 22:54:29.216403 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:29.216516 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 22:54:29.217875 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:29.217450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.0.1:52510)
0927 22:54:29.217635 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:29.217720 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:54:29.264657 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 14469
I20260927 22:54:29.278591 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:29.278014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.0.1:52520)
0927 22:54:29.278267 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:29.278379 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 22:54:29.279819 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:29.279359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.0.1:52528)
0927 22:54:29.279532 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:29.279626 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 22:54:29.282284 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 14602
Sep 27 22:54:29 dist-test-slave-df53 krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:54:29.298442 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:29.283010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.0.97.3:39533)
0927 22:54:29.283189 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:29.283196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:29.283216 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:54:29.283431 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:29.283437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:29.283514 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:29.283633 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:29.284859 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:29.285655 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:29.286979 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:29.287436 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:29.287573 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:29.292361 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:29.292372 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:29.292389 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:29.292432 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:29.295585 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:29.296428 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:29.296435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:29.296437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:29.296546 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:29.296962 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:29.296965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:29.296967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:29.297304 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:29.297550 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:29.297966 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 22:54:29.298144 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:54:29.299122 14725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:29.360627 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:29.360121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.0.1:52540)
0927 22:54:29.360366 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:29.360458 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 22:54:29.361825 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:29.361368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.0.1:52550)
0927 22:54:29.361558 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:29.361694 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 22:54:29.422286 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 14602
I20260927 22:54:29.441921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 14242
I20260927 22:54:29.443063 14304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:29.551924 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 14242
I20260927 22:54:29.574066 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35627
--webserver_interface=127.0.97.62
--webserver_port=34631
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:29.905453 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:29.905741 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:29.905839 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:29.913384 14785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:29.913481 14785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:29.913527 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:29.913583 14785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:29.913689 14785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:29.923497 14785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:29.924647 14785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:29.925999 14785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:29.947801 14792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:29.947896 14791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:29.947858 14794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:29.948621 14785 server_base.cc:1034] running on GCE node
I20260927 22:54:29.949450 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:29.950937 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:29.952201 14785 hybrid_clock.cc:648] HybridClock initialized: now 1790549669952150 us; error 57 us; skew 500 ppm
I20260927 22:54:29.952618 14785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:29 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:29.956148 14785 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:54:29.957702 14785 webserver.cc:460] Webserver started at http://127.0.97.62:34631/ using document root <none> and password file <none>
I20260927 22:54:29.958405 14785 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:29.958549 14785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:29.964270 14785 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:54:29.967084 14801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:29.968374 14785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:54:29.968551 14785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "0fec4059bbaf4faf9c395d38758ffeeb"
format_stamp: "Formatted at 2026-09-27 22:54:26 on dist-test-slave-df53"
server_key: "da5a6c328a6af2bcd9063be3fe118a69"
server_key_iv: "115d1685336cfe5389e64354d0accd80"
server_key_version: "encryptionkey@0"
I20260927 22:54:29.969605 14785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:30.006312 14785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:30.010499 14785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:30.010810 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:30.030547 14785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35627
I20260927 22:54:30.030570 14852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35627 every 8 connection(s)
I20260927 22:54:30.032039 14785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:30.037158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 14785
I20260927 22:54:30.038086 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42955
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35627
--webserver_port=38399
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:54:30.039157 14853 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:54:30.042436 14853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Bootstrap starting.
I20260927 22:54:30.061775 14853 log.cc:824] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:30.074163 14853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:54:30.074687 14853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Bootstrap complete.
I20260927 22:54:30.084259 14853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:30.085081 14853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fec4059bbaf4faf9c395d38758ffeeb, State: Initialized, Role: FOLLOWER
I20260927 22:54:30.085760 14853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:30.086011 14853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:30.086122 14853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:30.086287 14853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:54:30.089983 14853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:30.090468 14853 leader_election.cc:302] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fec4059bbaf4faf9c395d38758ffeeb; no voters:
I20260927 22:54:30.091120 14853 leader_election.cc:288] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:54:30.091252 14857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:54:30.092362 14857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 2 LEADER]: Becoming Leader. State: Replica: 0fec4059bbaf4faf9c395d38758ffeeb, State: Running, Role: LEADER
I20260927 22:54:30.092983 14857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:30.093371 14853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:30.095433 14859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fec4059bbaf4faf9c395d38758ffeeb. Latest consensus state: current_term: 2 leader_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } } }
I20260927 22:54:30.095775 14859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:30.096304 14858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } } }
I20260927 22:54:30.096570 14858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:30.102200 14864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:30.112264 14864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:30.114107 14864 catalog_manager.cc:1259] Loaded cluster ID: f207467b3e3b44e99de108cf392a224e
I20260927 22:54:30.114284 14864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:30.118428 14864 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 22:54:30.140377 14874 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:54:30.142297 14864 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 22:54:30.149189 14864 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:30.150488 14864 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Loaded TSK: 0
I20260927 22:54:30.151590 14864 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:54:30.393265 14855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:30.393574 14855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:30.393816 14855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:30.401170 14855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:30.401347 14855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:30.401575 14855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:54:30.411767 14855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38399
--tserver_master_addrs=127.0.97.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:30.412992 14855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:30.414335 14855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:30.428977 14880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:30.428939 14881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:30.430524 14883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:30.431253 14855 server_base.cc:1034] running on GCE node
I20260927 22:54:30.431978 14855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:30.433144 14855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:30.434394 14855 hybrid_clock.cc:648] HybridClock initialized: now 1790549670434322 us; error 62 us; skew 500 ppm
I20260927 22:54:30.434775 14855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:30 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:30.438819 14855 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:54:30.440528 14855 webserver.cc:460] Webserver started at http://127.0.97.1:38399/ using document root <none> and password file <none>
I20260927 22:54:30.441396 14855 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:30.441541 14855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:30.448418 14855 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:54:30.451753 14890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:30.453282 14855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:54:30.453438 14855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "3b509e932bed4262bc10cf887cf1f7f6"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-df53"
server_key: "808886ef724c1eae5baaca1d39cad6d3"
server_key_iv: "392d24168662b0d84466eb5d9ea3c8a4"
server_key_version: "encryptionkey@0"
I20260927 22:54:30.454742 14855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:30.475034 14855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:30.479094 14855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:30.479357 14855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:30.480438 14855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:30.482332 14897 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 22:54:30.487357 14897 ts_tablet_manager.cc:1852] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 22:54:30.488291 14897 ts_tablet_manager.cc:1905] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:30.495220 14897 ts_tablet_manager.cc:1918] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:30.495500 14897 ts_tablet_manager.cc:1939] T 512db29e8d5b44caae7db9dc8b476aac P 3b509e932bed4262bc10cf887cf1f7f6: Deleting consensus metadata
I20260927 22:54:30.495898 14855 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 22:54:30.495985 14855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.001s
I20260927 22:54:30.496119 14855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:30.496200 14855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:30.528046 14855 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42955
I20260927 22:54:30.528090 15003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42955 every 8 connection(s)
I20260927 22:54:30.529259 14855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:30.529903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 14855
I20260927 22:54:30.530678 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39339
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35627
--webserver_port=36257
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:54:30 dist-test-slave-df53 krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:30.555797 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:30.532497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:55533)
0927 22:54:30.533434 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:30.533504 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:30.536240 (+ 2736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:30.537566 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:30.537579 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:30.538156 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:30.538765 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:30.538793 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:30.540347 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:30.540355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:30.542602 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:30.542619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:30.543584 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:30.543593 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:30.543894 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:30.544617 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:30.544649 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:30.547233 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:30.551989 (+ 4756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:30.551995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:30.552006 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:30.552398 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:30.552916 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:30.552919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:30.552922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:30.553074 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:30.554075 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:30.554081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:30.554350 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 22:54:30.554557 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 22:54:30.554892 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":374,"threads_started":1}
I20260927 22:54:30.556617 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:30.532353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55533 (local address 127.0.97.62:35627)
0927 22:54:30.533065 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:30.533080 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:30.533655 (+ 575us) server_negotiation.cc:407] Connection header received
0927 22:54:30.536758 (+ 3103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:30.536771 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:30.537087 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:30.537369 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:30.539000 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:30.540192 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:30.542744 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:30.543363 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:30.543496 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:30.547520 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:30.547532 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:30.547552 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:30.547606 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:30.551852 (+ 4246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:30.552550 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:30.552562 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:30.552571 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:30.552795 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:30.553196 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:30.553203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:30.553210 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:30.553945 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:30.554165 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:30.554847 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 22:54:30.555331 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":265,"threads_started":1}
I20260927 22:54:30.559419 15004 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35627
I20260927 22:54:30.559751 15004 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:30.560467 15004 heartbeater.cc:511] Master 127.0.97.62:35627 requested a full tablet report, sending...
I20260927 22:54:30.562597 14818 ts_manager.cc:194] Registered new tserver with Master: 3b509e932bed4262bc10cf887cf1f7f6 (127.0.97.1:42955)
I20260927 22:54:30.564250 14818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55533
W20260927 22:54:30.847952 15008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:30.848224 15008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:30.848385 15008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:30.855252 15008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:30.855376 15008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:30.855561 15008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:54:30.864907 15008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36257
--tserver_master_addrs=127.0.97.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:30.866075 15008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:30.867283 15008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:30.879638 15013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:30.879688 15014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:30.879603 15016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:30.880657 15008 server_base.cc:1034] running on GCE node
I20260927 22:54:30.881194 15008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:30.882063 15008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:30.883252 15008 hybrid_clock.cc:648] HybridClock initialized: now 1790549670883211 us; error 29 us; skew 500 ppm
I20260927 22:54:30.883512 15008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:30 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:30.886445 15008 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:54:30.887825 15008 webserver.cc:460] Webserver started at http://127.0.97.2:36257/ using document root <none> and password file <none>
I20260927 22:54:30.888456 15008 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:30.888599 15008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:30.893821 15008 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:54:30.896639 15023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:30.897894 15008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:30.898070 15008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "e2a47bc0e167486f9ef0ce77abb268ea"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-df53"
server_key: "a46247804e8eff0de0d83c13541850ed"
server_key_iv: "0718bf04f9cb24bfd08a791e43171327"
server_key_version: "encryptionkey@0"
I20260927 22:54:30.899000 15008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:30.913546 15008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:30.917868 15008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:30.918128 15008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:30.919013 15008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:30.920230 15008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:30.920321 15008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:30.920464 15008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:30.920553 15008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:30.952334 15008 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39339
I20260927 22:54:30.952415 15135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39339 every 8 connection(s)
I20260927 22:54:30.953715 15008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:30.963971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 15008
I20260927 22:54:30.964656 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39533
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35627
--webserver_port=34133
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:54:30 dist-test-slave-df53 krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:30.973182 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:30.956231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46787 (local address 127.0.97.62:35627)
0927 22:54:30.956395 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:30.956399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:30.957440 (+ 1041us) server_negotiation.cc:407] Connection header received
0927 22:54:30.959518 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:30.959523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:30.959640 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:30.959750 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:30.961259 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:30.962053 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:30.963645 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:30.964073 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:30.964220 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:30.967590 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:30.967598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:30.967607 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:30.967639 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:30.970197 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:30.970897 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:30.970903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:30.970907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:30.971039 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:30.971457 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:30.971463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:30.971467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:30.971726 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:30.971892 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:30.972550 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 22:54:30.972823 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:54:30.974148 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:30.956606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:46787)
0927 22:54:30.957274 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:30.957310 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:30.959221 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:30.959996 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:30.960004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:30.960552 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:30.961088 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:30.961105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:30.962201 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:30.962213 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:30.963497 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:30.963508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:30.964303 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:30.964311 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:30.964538 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:30.965221 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:30.965249 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:30.967371 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:30.970351 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:30.970359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:30.970374 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:30.970763 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:30.971153 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:30.971158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:30.971162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:30.971338 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:30.971851 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:30.971861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:30.972185 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 22:54:30.972440 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 22:54:30.972941 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":277,"threads_started":1}
I20260927 22:54:30.975564 15136 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35627
I20260927 22:54:30.975884 15136 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:30.976713 15136 heartbeater.cc:511] Master 127.0.97.62:35627 requested a full tablet report, sending...
I20260927 22:54:30.978488 14818 ts_manager.cc:194] Registered new tserver with Master: e2a47bc0e167486f9ef0ce77abb268ea (127.0.97.2:39339)
I20260927 22:54:30.979274 14818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46787
W20260927 22:54:31.289942 15139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:31.290186 15139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:31.290334 15139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:31.297427 15139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:31.297518 15139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:31.297726 15139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:54:31.306967 15139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34133
--tserver_master_addrs=127.0.97.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:31.308024 15139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:31.309170 15139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:31.321890 15147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:31.322233 15145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:31.321723 15144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:31.322829 15139 server_base.cc:1034] running on GCE node
I20260927 22:54:31.323455 15139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:31.324391 15139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:31.325624 15139 hybrid_clock.cc:648] HybridClock initialized: now 1790549671325539 us; error 45 us; skew 500 ppm
I20260927 22:54:31.325947 15139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:31 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:31.328920 15139 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:54:31.330500 15139 webserver.cc:460] Webserver started at http://127.0.97.3:34133/ using document root <none> and password file <none>
I20260927 22:54:31.331146 15139 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:31.331296 15139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:31.336071 15139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 22:54:31.338557 15154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:31.339797 15139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:54:31.339946 15139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "e18a873f084549f9ad7b5c384818ba15"
format_stamp: "Formatted at 2026-09-27 22:54:28 on dist-test-slave-df53"
server_key: "d9b3f9e7e58a4906292f48255e9cfd15"
server_key_iv: "a87608d04cb9c8473e766b491257876b"
server_key_version: "encryptionkey@0"
I20260927 22:54:31.340870 15139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:31.357717 15139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:31.361754 15139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:31.362016 15139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:31.362937 15139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:31.364190 15139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:31.364302 15139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:31.364435 15139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:31.364526 15139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:31.397779 15139 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39533
I20260927 22:54:31.397871 15266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39533 every 8 connection(s)
I20260927 22:54:31.399174 15139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:31.407018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 15139
Sep 27 22:54:31 dist-test-slave-df53 krb5kdc[14229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:31.421356 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.401527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53693 (local address 127.0.97.62:35627)
0927 22:54:31.401759 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.401764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.402965 (+ 1201us) server_negotiation.cc:407] Connection header received
0927 22:54:31.404752 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.404757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.404847 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.404966 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:31.406711 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.407525 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.409581 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.410362 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.410508 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.414379 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:31.414384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:31.414387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:31.414421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:31.418011 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:31.418882 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:31.418889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:31.418894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:31.419066 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:31.419555 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:31.419558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:31.419560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:31.419958 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:31.420073 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.420899 (+ 826us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.421120 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 22:54:31.422737 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.402107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:53693)
0927 22:54:31.402783 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:31.402824 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:31.404500 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:31.405183 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:31.405190 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:31.405739 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:31.406503 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.406520 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.407714 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.407719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.409441 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.409452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.410562 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.410567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.410755 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.411438 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:31.411466 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:31.414166 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:31.418131 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:31.418141 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:31.418159 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:31.418738 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:31.419185 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:31.419190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:31.419193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:31.419440 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:31.420088 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:31.420098 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:31.420498 (+ 400us) client_negotiation.cc:769] Sending connection context
0927 22:54:31.420806 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 22:54:31.421333 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":261,"threads_started":1}
I20260927 22:54:31.424134 15267 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35627
I20260927 22:54:31.424479 15267 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:31.424501 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.407897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.0.97.62:35627)
0927 22:54:31.408400 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.408406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.408843 (+ 437us) server_negotiation.cc:407] Connection header received
0927 22:54:31.408982 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.408988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.409057 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.409170 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:31.411019 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.412158 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.413940 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.414552 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.414640 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.418007 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:31.418012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:31.418015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:31.418043 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:31.421399 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:31.422145 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:31.422150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:31.422153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:31.422285 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:31.422646 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:31.422651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:31.422654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:31.423361 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:31.423566 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.423913 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.424115 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":256,"threads_started":1}
I20260927 22:54:31.425310 15267 heartbeater.cc:511] Master 127.0.97.62:35627 requested a full tablet report, sending...
I20260927 22:54:31.426726 14818 ts_manager.cc:194] Registered new tserver with Master: e18a873f084549f9ad7b5c384818ba15 (127.0.97.3:39533)
I20260927 22:54:31.427495 14818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53693
I20260927 22:54:31.430943 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:54:31.437368 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.431856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.0.97.62:35627)
0927 22:54:31.432073 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.432079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.432343 (+ 264us) server_negotiation.cc:407] Connection header received
0927 22:54:31.432576 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.432580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.432693 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.432831 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:54:31.433249 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.433982 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.435377 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.435782 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.435923 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.436380 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:54:31.436818 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:54:31.436983 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.437062 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.437131 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:54:31.438315 14818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:54:31.445570 14818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:54:31.467631 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.457546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.97.3:39533)
0927 22:54:31.458378 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.458442 (+ 64us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.459165 (+ 723us) server_negotiation.cc:407] Connection header received
0927 22:54:31.459510 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.459521 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.459789 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.459957 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:31.461097 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.462532 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.465534 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.466466 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.466608 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.466842 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.467133 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.467286 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":489,"threads_started":1}
I20260927 22:54:31.467772 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.457279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39533 (local address 127.0.0.1:53548)
0927 22:54:31.459047 (+ 1768us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:31.459072 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:31.459265 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:31.460276 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:31.460296 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:31.460346 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:31.460863 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.460880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.462684 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.462692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.465386 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.465397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.466675 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.466681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.466897 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.466957 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 22:54:31.467097 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 22:54:31.467268 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1603,"mutex_wait_us":241,"spinlock_wait_cycles":20480,"thread_start_us":1416,"threads_started":1}
I20260927 22:54:31.468371 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.456718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.0.1:52570)
0927 22:54:31.459043 (+ 2325us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:31.459071 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:31.459257 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:31.460042 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:31.460050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:31.460082 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:31.460731 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.460746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.462782 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.462787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.465384 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.465397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.467496 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.467502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.467678 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.467712 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:54:31.467808 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:54:31.467937 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2115,"mutex_wait_us":109,"spinlock_wait_cycles":2944,"thread_start_us":2568,"threads_started":1}
I20260927 22:54:31.469163 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.456957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.0.97.1:42955)
0927 22:54:31.458661 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.458670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.459171 (+ 501us) server_negotiation.cc:407] Connection header received
0927 22:54:31.459457 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.459508 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.459790 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.460161 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:31.461049 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.462632 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.465575 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.467002 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.467130 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.467339 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.468680 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.468841 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1538,"thread_start_us":1374,"threads_started":1}
I20260927 22:54:31.470615 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.460220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.0.97.2:39339)
0927 22:54:31.461850 (+ 1630us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.461857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.461885 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:54:31.461967 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.461973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.462166 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.462392 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:31.463290 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.464881 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.467311 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.469485 (+ 2174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.469662 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.469903 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.470109 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.470266 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1474,"thread_start_us":1361,"threads_started":1}
I20260927 22:54:31.471387 14939 tablet_service.cc:1467] Processing CreateTablet for tablet 39b0427c3ebb4674b44653efffbe9177 (DEFAULT_TABLE table=test-table [id=c8023c1d03594228bd1d4febe934f985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:31.471645 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 39b0427c3ebb4674b44653efffbe9177 (DEFAULT_TABLE table=test-table [id=c8023c1d03594228bd1d4febe934f985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:31.472574 14939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b0427c3ebb4674b44653efffbe9177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:31.473006 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b0427c3ebb4674b44653efffbe9177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:31.477839 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.459633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39339 (local address 127.0.0.1:51420)
0927 22:54:31.461244 (+ 1611us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:31.461260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:31.461357 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:31.462396 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:31.462400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:31.462428 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:31.463076 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.463094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.465016 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.465022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.467137 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.467150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.469729 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.469735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.469904 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.469943 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:54:31.470055 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 22:54:31.470175 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1506,"mutex_wait_us":1,"thread_start_us":784,"threads_started":1}
I20260927 22:54:31.481001 15071 tablet_service.cc:1467] Processing CreateTablet for tablet 39b0427c3ebb4674b44653efffbe9177 (DEFAULT_TABLE table=test-table [id=c8023c1d03594228bd1d4febe934f985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:31.482178 15283 tablet_bootstrap.cc:492] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: Bootstrap starting.
I20260927 22:54:31.482275 15071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b0427c3ebb4674b44653efffbe9177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:31.485930 15284 tablet_bootstrap.cc:492] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: Bootstrap starting.
I20260927 22:54:31.486726 15283 tablet_bootstrap.cc:654] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:31.487953 15283 log.cc:824] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:31.488759 15284 tablet_bootstrap.cc:654] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:31.489915 15284 log.cc:824] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:31.491060 15283 tablet_bootstrap.cc:492] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: No bootstrap required, opened a new log
I20260927 22:54:31.491330 15283 ts_tablet_manager.cc:1397] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 22:54:31.492292 15284 tablet_bootstrap.cc:492] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: No bootstrap required, opened a new log
I20260927 22:54:31.492547 15284 ts_tablet_manager.cc:1397] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260927 22:54:31.497251 15287 tablet_bootstrap.cc:492] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: Bootstrap starting.
I20260927 22:54:31.500290 15283 raft_consensus.cc:348] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } }
I20260927 22:54:31.500483 15287 tablet_bootstrap.cc:654] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:31.500638 15283 raft_consensus.cc:374] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:31.500751 15283 raft_consensus.cc:729] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b509e932bed4262bc10cf887cf1f7f6, State: Initialized, Role: FOLLOWER
I20260927 22:54:31.501437 15283 consensus_queue.cc:260] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } }
I20260927 22:54:31.501744 15287 log.cc:824] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:31.501690 15284 raft_consensus.cc:348] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } }
I20260927 22:54:31.502143 15284 raft_consensus.cc:374] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:31.502333 15284 raft_consensus.cc:729] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18a873f084549f9ad7b5c384818ba15, State: Initialized, Role: FOLLOWER
I20260927 22:54:31.502511 15004 heartbeater.cc:503] Master 127.0.97.62:35627 was elected leader, sending a full tablet report...
I20260927 22:54:31.503026 15284 consensus_queue.cc:260] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } }
I20260927 22:54:31.505326 15284 ts_tablet_manager.cc:1428] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260927 22:54:31.505537 15267 heartbeater.cc:503] Master 127.0.97.62:35627 was elected leader, sending a full tablet report...
I20260927 22:54:31.507337 15283 ts_tablet_manager.cc:1428] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260927 22:54:31.508188 15287 tablet_bootstrap.cc:492] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: No bootstrap required, opened a new log
I20260927 22:54:31.508435 15287 ts_tablet_manager.cc:1397] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260927 22:54:31.515201 15287 raft_consensus.cc:348] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } }
I20260927 22:54:31.515587 15287 raft_consensus.cc:374] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:31.515691 15287 raft_consensus.cc:729] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a47bc0e167486f9ef0ce77abb268ea, State: Initialized, Role: FOLLOWER
I20260927 22:54:31.516242 15287 consensus_queue.cc:260] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } }
I20260927 22:54:31.517107 15136 heartbeater.cc:503] Master 127.0.97.62:35627 was elected leader, sending a full tablet report...
I20260927 22:54:31.517706 15287 ts_tablet_manager.cc:1428] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: Time spent starting tablet: real 0.009s user 0.010s sys 0.001s
W20260927 22:54:31.531623 15005 tablet.cc:2369] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:54:31.554554 15290 raft_consensus.cc:482] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:54:31.554842 15290 raft_consensus.cc:504] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } }
I20260927 22:54:31.556384 15290 leader_election.cc:288] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b509e932bed4262bc10cf887cf1f7f6 (127.0.97.1:42955), e2a47bc0e167486f9ef0ce77abb268ea (127.0.97.2:39339)
I20260927 22:54:31.563047 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.556751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42955 (local address 127.0.97.3:47079)
0927 22:54:31.556961 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:31.556982 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:31.557113 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:31.557471 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:31.557475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:31.557503 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:31.557781 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.557800 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.559148 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.559152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.561112 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.561126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.562405 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.562410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.562539 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.562623 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 22:54:31.562695 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 22:54:31.562782 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":26}
I20260927 22:54:31.563172 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.556796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47079 (local address 127.0.97.1:42955)
0927 22:54:31.556998 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.557003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.557025 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:54:31.557265 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.557269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.557326 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.557435 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:31.557960 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.558993 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.561271 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.562267 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.562397 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.562487 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.562738 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.562906 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:54:31.564143 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.557362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46943 (local address 127.0.97.2:39339)
0927 22:54:31.557644 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.557653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.558169 (+ 516us) server_negotiation.cc:407] Connection header received
0927 22:54:31.558249 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.558255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.558336 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.558433 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:31.558954 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.559730 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.561923 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.563139 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.563269 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.563379 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.563783 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.563907 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:54:31.564267 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.557199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39339 (local address 127.0.97.3:46943)
0927 22:54:31.558061 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:31.558074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:31.558167 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:31.558488 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:31.558492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:31.558522 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:31.558794 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.558802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.559882 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.559889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.561776 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:31.561787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.563490 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.563495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:31.563607 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.563628 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:54:31.563757 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 22:54:31.563888 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":50,"thread_start_us":509,"threads_started":1}
I20260927 22:54:31.564131 14959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b0427c3ebb4674b44653efffbe9177" candidate_uuid: "e18a873f084549f9ad7b5c384818ba15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b509e932bed4262bc10cf887cf1f7f6" is_pre_election: true
I20260927 22:54:31.564684 14959 raft_consensus.cc:2454] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e18a873f084549f9ad7b5c384818ba15 in term 0.
I20260927 22:54:31.565236 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b0427c3ebb4674b44653efffbe9177" candidate_uuid: "e18a873f084549f9ad7b5c384818ba15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" is_pre_election: true
I20260927 22:54:31.565433 15156 leader_election.cc:302] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b509e932bed4262bc10cf887cf1f7f6, e18a873f084549f9ad7b5c384818ba15; no voters:
I20260927 22:54:31.565775 15091 raft_consensus.cc:2454] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e18a873f084549f9ad7b5c384818ba15 in term 0.
I20260927 22:54:31.565866 15290 raft_consensus.cc:2781] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:54:31.566036 15290 raft_consensus.cc:482] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:54:31.566147 15290 raft_consensus.cc:3037] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:31.568360 15290 raft_consensus.cc:504] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } }
I20260927 22:54:31.568924 15290 leader_election.cc:288] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [CANDIDATE]: Term 1 election: Requested vote from peers 3b509e932bed4262bc10cf887cf1f7f6 (127.0.97.1:42955), e2a47bc0e167486f9ef0ce77abb268ea (127.0.97.2:39339)
I20260927 22:54:31.569330 14959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b0427c3ebb4674b44653efffbe9177" candidate_uuid: "e18a873f084549f9ad7b5c384818ba15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b509e932bed4262bc10cf887cf1f7f6"
I20260927 22:54:31.569409 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b0427c3ebb4674b44653efffbe9177" candidate_uuid: "e18a873f084549f9ad7b5c384818ba15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a47bc0e167486f9ef0ce77abb268ea"
I20260927 22:54:31.569521 14959 raft_consensus.cc:3037] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:31.569636 15091 raft_consensus.cc:3037] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:31.572412 14959 raft_consensus.cc:2454] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e18a873f084549f9ad7b5c384818ba15 in term 1.
I20260927 22:54:31.572683 15091 raft_consensus.cc:2454] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e18a873f084549f9ad7b5c384818ba15 in term 1.
I20260927 22:54:31.572885 15156 leader_election.cc:302] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b509e932bed4262bc10cf887cf1f7f6, e18a873f084549f9ad7b5c384818ba15; no voters:
I20260927 22:54:31.573233 15290 raft_consensus.cc:2781] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:31.573798 15290 raft_consensus.cc:686] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 1 LEADER]: Becoming Leader. State: Replica: e18a873f084549f9ad7b5c384818ba15, State: Running, Role: LEADER
I20260927 22:54:31.574384 15290 consensus_queue.cc:237] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } }
I20260927 22:54:31.580047 14817 catalog_manager.cc:5495] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 reported cstate change: term changed from 0 to 1, leader changed from <none> to e18a873f084549f9ad7b5c384818ba15 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e18a873f084549f9ad7b5c384818ba15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e18a873f084549f9ad7b5c384818ba15" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39533 } health_report { overall_health: HEALTHY } } }
I20260927 22:54:31.618840 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.613557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.0.97.3:39533)
0927 22:54:31.613760 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.613766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.613833 (+ 67us) server_negotiation.cc:407] Connection header received
0927 22:54:31.614074 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.614078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.614149 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.614291 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:54:31.614642 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.615354 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.616777 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.617175 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.617306 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.617822 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:54:31.618258 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:54:31.618411 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.618514 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.618586 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:54:31.626441 15091 raft_consensus.cc:1261] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea [term 1 FOLLOWER]: Refusing update from remote peer e18a873f084549f9ad7b5c384818ba15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:31.626731 14959 raft_consensus.cc:1261] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6 [term 1 FOLLOWER]: Refusing update from remote peer e18a873f084549f9ad7b5c384818ba15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:31.627333 15293 consensus_queue.cc:1035] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a47bc0e167486f9ef0ce77abb268ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:31.627681 15290 consensus_queue.cc:1035] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b509e932bed4262bc10cf887cf1f7f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:31.639050 15295 mvcc.cc:205] Tried to move back new op lower bound from 7334091454973218816 to 7334091454772633600. Current Snapshot: MvccSnapshot[applied={T|T < 7334091454973218816}]
I20260927 22:54:31.639762 15297 mvcc.cc:205] Tried to move back new op lower bound from 7334091454973218816 to 7334091454772633600. Current Snapshot: MvccSnapshot[applied={T|T < 7334091454973218816}]
I20260927 22:54:31.650878 14817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:54:31.651212 14817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:54:31.654529 14817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Sending DeleteTablet for 3 replicas of tablet 39b0427c3ebb4674b44653efffbe9177
I20260927 22:54:31.655645 14939 tablet_service.cc:1514] Processing DeleteTablet for tablet 39b0427c3ebb4674b44653efffbe9177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52570
I20260927 22:54:31.655669 15071 tablet_service.cc:1514] Processing DeleteTablet for tablet 39b0427c3ebb4674b44653efffbe9177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51420
I20260927 22:54:31.655699 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet 39b0427c3ebb4674b44653efffbe9177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53548
I20260927 22:54:31.655965 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 14855
I20260927 22:54:31.670773 15307 tablet_replica.cc:330] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: stopping tablet replica
I20260927 22:54:31.671079 15305 tablet_replica.cc:330] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: stopping tablet replica
I20260927 22:54:31.671274 15306 tablet_replica.cc:330] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: stopping tablet replica
I20260927 22:54:31.671464 15307 raft_consensus.cc:2229] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:54:31.671481 15305 raft_consensus.cc:2229] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:54:31.671674 15306 raft_consensus.cc:2229] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:54:31.672008 15305 raft_consensus.cc:2258] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:31.672076 15306 raft_consensus.cc:2258] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:31.672367 15307 raft_consensus.cc:2258] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:31.674229 15305 ts_tablet_manager.cc:1905] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:31.674254 15306 ts_tablet_manager.cc:1905] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:31.679706 15307 ts_tablet_manager.cc:1905] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:31.680342 15305 ts_tablet_manager.cc:1918] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:31.680474 15305 log.cc:1196] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/39b0427c3ebb4674b44653efffbe9177
I20260927 22:54:31.681017 15305 ts_tablet_manager.cc:1939] T 39b0427c3ebb4674b44653efffbe9177 P 3b509e932bed4262bc10cf887cf1f7f6: Deleting consensus metadata
I20260927 22:54:31.682265 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.657643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.0.97.1:42955)
0927 22:54:31.657845 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.657854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.657875 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:54:31.657942 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.657950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.658077 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.658223 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:31.661237 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.662465 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.664502 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.665174 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.665296 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.668895 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:31.668910 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:31.668927 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:31.668972 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:31.678019 (+ 9047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:31.678895 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:31.678904 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:31.678907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:31.679061 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:31.679537 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:31.679542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:31.679546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:31.679961 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:31.680244 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.681693 (+ 1449us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.681901 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:54:31.682893 14803 catalog_manager.cc:4841] TS 3b509e932bed4262bc10cf887cf1f7f6 (127.0.97.1:42955): tablet 39b0427c3ebb4674b44653efffbe9177 (table test-table [id=c8023c1d03594228bd1d4febe934f985]) successfully deleted
I20260927 22:54:31.683118 14999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:31.688316 15306 ts_tablet_manager.cc:1918] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:31.688325 15307 ts_tablet_manager.cc:1918] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:31.688468 15306 log.cc:1196] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/39b0427c3ebb4674b44653efffbe9177
I20260927 22:54:31.688628 15307 log.cc:1196] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/39b0427c3ebb4674b44653efffbe9177
I20260927 22:54:31.689029 15306 ts_tablet_manager.cc:1939] T 39b0427c3ebb4674b44653efffbe9177 P e2a47bc0e167486f9ef0ce77abb268ea: Deleting consensus metadata
I20260927 22:54:31.689293 15307 ts_tablet_manager.cc:1939] T 39b0427c3ebb4674b44653efffbe9177 P e18a873f084549f9ad7b5c384818ba15: Deleting consensus metadata
I20260927 22:54:31.690758 14803 catalog_manager.cc:4841] TS e2a47bc0e167486f9ef0ce77abb268ea (127.0.97.2:39339): tablet 39b0427c3ebb4674b44653efffbe9177 (table test-table [id=c8023c1d03594228bd1d4febe934f985]) successfully deleted
I20260927 22:54:31.690759 14802 catalog_manager.cc:4841] TS e18a873f084549f9ad7b5c384818ba15 (127.0.97.3:39533): tablet 39b0427c3ebb4674b44653efffbe9177 (table test-table [id=c8023c1d03594228bd1d4febe934f985]) successfully deleted
I20260927 22:54:31.779642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 14855
I20260927 22:54:31.798281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 15008
I20260927 22:54:31.814061 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.799200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.0.97.2:39339)
0927 22:54:31.799375 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.799379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.799397 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:54:31.799515 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.799519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.799583 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.799673 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:31.801389 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.802504 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.804372 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.804780 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.804879 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.808341 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:31.808354 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:31.808369 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:31.808401 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:31.811320 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:31.812144 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:31.812151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:31.812153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:31.812265 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:31.812659 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:31.812662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:31.812664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:31.812985 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:31.813189 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.813545 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.813757 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:54:31.814781 15131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:31.902513 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 15008
I20260927 22:54:31.925455 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 15139
I20260927 22:54:31.939616 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:31.926292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.0.97.3:39533)
0927 22:54:31.926504 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:31.926509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:31.926529 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:54:31.926736 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:31.926741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:31.926800 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:31.926915 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:31.928234 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.929062 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.930502 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:31.930977 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:31.931116 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:31.933683 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:31.933696 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:31.933710 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:31.933744 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:31.936676 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:31.937396 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:31.937407 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:31.937411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:31.937575 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:31.937992 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:31.937998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:31.938001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:31.938345 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:31.938681 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:31.939011 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 22:54:31.939233 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:54:31.940448 15262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:32.061944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 15139
I20260927 22:54:32.080902 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 14785
I20260927 22:54:32.082033 14848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:32.193626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 14785
I20260927 22:54:32.214699 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35627
--webserver_interface=127.0.97.62
--webserver_port=34631
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:32.530964 15312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:32.531286 15312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:32.531450 15312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:32.538516 15312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:32.538658 15312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:32.538766 15312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:32.538878 15312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:32.538966 15312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:32.548378 15312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:32.550105 15312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:32.551391 15312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:32.562772 15320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:32.562773 15318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:32.562876 15317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:32.564265 15312 server_base.cc:1034] running on GCE node
I20260927 22:54:32.564990 15312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:32.566396 15312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:32.567636 15312 hybrid_clock.cc:648] HybridClock initialized: now 1790549672567593 us; error 49 us; skew 500 ppm
I20260927 22:54:32.567979 15312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:32 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:32.571132 15312 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:54:32.572593 15312 webserver.cc:460] Webserver started at http://127.0.97.62:34631/ using document root <none> and password file <none>
I20260927 22:54:32.573276 15312 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:32.573431 15312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:32.578759 15312 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:54:32.581518 15327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:32.582880 15312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:54:32.583061 15312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "0fec4059bbaf4faf9c395d38758ffeeb"
format_stamp: "Formatted at 2026-09-27 22:54:26 on dist-test-slave-df53"
server_key: "da5a6c328a6af2bcd9063be3fe118a69"
server_key_iv: "115d1685336cfe5389e64354d0accd80"
server_key_version: "encryptionkey@0"
I20260927 22:54:32.584019 15312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:32.596040 15312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:32.600476 15312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:32.600740 15312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:32.617798 15312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35627
I20260927 22:54:32.617823 15378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35627 every 8 connection(s)
I20260927 22:54:32.619197 15312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:32.624337 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 15312
I20260927 22:54:32.625316 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42955
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35627
--webserver_port=38399
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:54:32.626499 15379 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:54:32.631548 15379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Bootstrap starting.
I20260927 22:54:32.651444 15379 log.cc:824] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:32.667186 15379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:54:32.667685 15379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb: Bootstrap complete.
I20260927 22:54:32.676569 15379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:32.677436 15379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fec4059bbaf4faf9c395d38758ffeeb, State: Initialized, Role: FOLLOWER
I20260927 22:54:32.678123 15379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:32.678367 15379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:32.678488 15379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:32.678674 15379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:54:32.682374 15379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:32.682863 15379 leader_election.cc:302] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fec4059bbaf4faf9c395d38758ffeeb; no voters:
I20260927 22:54:32.683554 15379 leader_election.cc:288] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:54:32.683682 15383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:54:32.684751 15383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [term 3 LEADER]: Becoming Leader. State: Replica: 0fec4059bbaf4faf9c395d38758ffeeb, State: Running, Role: LEADER
I20260927 22:54:32.685279 15383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } }
I20260927 22:54:32.685714 15379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:32.687829 15384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } } }
I20260927 22:54:32.688169 15384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:32.688758 15385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fec4059bbaf4faf9c395d38758ffeeb. Latest consensus state: current_term: 3 leader_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec4059bbaf4faf9c395d38758ffeeb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35627 } } }
I20260927 22:54:32.689018 15385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fec4059bbaf4faf9c395d38758ffeeb [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:32.694324 15388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:32.704396 15388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:32.706267 15388 catalog_manager.cc:1259] Loaded cluster ID: f207467b3e3b44e99de108cf392a224e
I20260927 22:54:32.706383 15388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 22:54:32.708920 15388 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f49b5af99ad google::LogMessage::Fail() at ??:0
@ 0x7f49b5afc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f49b5af94de google::LogMessage::Flush() at ??:0
@ 0x7f49b5afaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f49cb52c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f49cb529976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f49cb5b1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f49cb5b1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f49b8745345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f49b874891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f49b8748799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f49b8723eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f49bb9fd6db start_thread at ??:0
@ 0x7f49b499361f clone at ??:0
W20260927 22:54:33.015897 15381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:33.016156 15381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:33.016248 15381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:33.023509 15381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:33.023607 15381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:33.023716 15381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:54:33.033200 15381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38399
--tserver_master_addrs=127.0.97.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:33.034343 15381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:33.036163 15381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:33.049455 15423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:33.049518 15421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:33.049423 15420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:33.050256 15381 server_base.cc:1034] running on GCE node
I20260927 22:54:33.050836 15381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:33.051725 15381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:33.052956 15381 hybrid_clock.cc:648] HybridClock initialized: now 1790549673052912 us; error 67 us; skew 500 ppm
I20260927 22:54:33.053251 15381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:33 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:33.056303 15381 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:54:33.057773 15381 webserver.cc:460] Webserver started at http://127.0.97.1:38399/ using document root <none> and password file <none>
I20260927 22:54:33.058355 15381 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:33.058463 15381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:33.063493 15381 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 22:54:33.066241 15430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:33.067549 15381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:54:33.067710 15381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "3b509e932bed4262bc10cf887cf1f7f6"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-df53"
server_key: "808886ef724c1eae5baaca1d39cad6d3"
server_key_iv: "392d24168662b0d84466eb5d9ea3c8a4"
server_key_version: "encryptionkey@0"
I20260927 22:54:33.068732 15381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:33.085058 15381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:33.089159 15381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:33.089380 15381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:33.090345 15381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:33.091576 15381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:33.091668 15381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:33.091743 15381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:33.091810 15381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:33.124814 15381 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42955
I20260927 22:54:33.124861 15542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42955 every 8 connection(s)
I20260927 22:54:33.126257 15381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:33.130183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 15381
I20260927 22:54:33.130955 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39339
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35627
--webserver_port=36257
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:54:33.131332 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:33.129041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:37459)
0927 22:54:33.129881 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:33.130121 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":286,"threads_started":1}
W20260927 22:54:33.132066 15543 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35627 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35627: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
I20260927 22:54:33.132977 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:33.132567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:60977)
0927 22:54:33.132739 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:33.132826 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 22:54:33.133944 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:33.133498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:46345)
0927 22:54:33.133680 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:33.133773 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:54:33.134182 15543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:54:33.463647 15546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:33.463924 15546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:33.464074 15546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:33.471200 15546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:33.471319 15546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:33.471505 15546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:54:33.481112 15546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36257
--tserver_master_addrs=127.0.97.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:33.482292 15546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:33.483487 15546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:33.496177 15552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:33.496237 15551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:33.496263 15554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:33.496893 15546 server_base.cc:1034] running on GCE node
I20260927 22:54:33.497488 15546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:33.498358 15546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:33.499585 15546 hybrid_clock.cc:648] HybridClock initialized: now 1790549673499543 us; error 54 us; skew 500 ppm
I20260927 22:54:33.499909 15546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:33 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:33.502945 15546 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:54:33.504230 15546 webserver.cc:460] Webserver started at http://127.0.97.2:36257/ using document root <none> and password file <none>
I20260927 22:54:33.504846 15546 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:33.504979 15546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:33.509995 15546 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:54:33.512552 15561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:33.513849 15546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:33.514019 15546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "e2a47bc0e167486f9ef0ce77abb268ea"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-df53"
server_key: "a46247804e8eff0de0d83c13541850ed"
server_key_iv: "0718bf04f9cb24bfd08a791e43171327"
server_key_version: "encryptionkey@0"
I20260927 22:54:33.514953 15546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:33.526394 15546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:33.530117 15546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:33.530375 15546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:33.531275 15546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:33.532518 15546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:33.532635 15546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:33.532765 15546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:33.532853 15546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:33.565229 15546 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39339
I20260927 22:54:33.565346 15673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39339 every 8 connection(s)
I20260927 22:54:33.566726 15546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:33.572196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 15546
I20260927 22:54:33.572995 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39533
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35627
--webserver_port=34133
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:54:33.571744 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:33.569688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:37601)
0927 22:54:33.570393 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:33.570591 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":261,"threads_started":1}
W20260927 22:54:33.574486 15674 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35627 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35627: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
I20260927 22:54:33.575476 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:33.575072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:42829)
0927 22:54:33.575247 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:33.575330 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 22:54:33.576874 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:33.576316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:55275)
0927 22:54:33.576638 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:33.576728 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
W20260927 22:54:33.577263 15674 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:54:33.889449 15677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:33.889832 15677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:33.890019 15677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:33.897132 15677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:33.897282 15677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:33.897480 15677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:54:33.907174 15677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34133
--tserver_master_addrs=127.0.97.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:33.908471 15677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:33.909771 15677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:33.922956 15685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:33.922977 15683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:33.923012 15682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:33.924234 15677 server_base.cc:1034] running on GCE node
I20260927 22:54:33.924786 15677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:33.925774 15677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:33.926999 15677 hybrid_clock.cc:648] HybridClock initialized: now 1790549673926934 us; error 49 us; skew 500 ppm
I20260927 22:54:33.927304 15677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:33 dist-test-slave-df53 krb5kdc[14229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:33.930326 15677 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:54:33.931633 15677 webserver.cc:460] Webserver started at http://127.0.97.3:34133/ using document root <none> and password file <none>
I20260927 22:54:33.932241 15677 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:33.932335 15677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:33.937398 15677 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:54:33.940065 15692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:33.941382 15677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:54:33.941517 15677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "e18a873f084549f9ad7b5c384818ba15"
format_stamp: "Formatted at 2026-09-27 22:54:28 on dist-test-slave-df53"
server_key: "d9b3f9e7e58a4906292f48255e9cfd15"
server_key_iv: "a87608d04cb9c8473e766b491257876b"
server_key_version: "encryptionkey@0"
I20260927 22:54:33.942493 15677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:33.970000 15677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:33.974442 15677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:33.974689 15677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:33.975754 15677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:33.977316 15677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:33.977424 15677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:33.977497 15677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:33.977556 15677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:34.010710 15677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39533
I20260927 22:54:34.010805 15804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39533 every 8 connection(s)
I20260927 22:54:34.012146 15677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:34.016105 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 15677
I20260927 22:54:34.017102 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:34.015046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:59337)
0927 22:54:34.015783 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:34.016066 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":263,"threads_started":1}
W20260927 22:54:34.017817 15805 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35627 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35627: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
I20260927 22:54:34.018697 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:34.018398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:44881)
0927 22:54:34.018533 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:34.018595 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:54:34.019590 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:34.019273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:46943)
0927 22:54:34.019424 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:34.019491 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 22:54:34.019831 15805 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:54:34.135416 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:34.134685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:48319)
0927 22:54:34.135109 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:34.135188 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":209,"threads_started":1}
I20260927 22:54:34.578583 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:34.577816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:53909)
0927 22:54:34.578261 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:34.578359 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260927 22:54:35.021051 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:35.020315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:57891)
0927 22:54:35.020748 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:35.020829 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":220,"threads_started":1}
I20260927 22:54:35.137071 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:35.136321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:49129)
0927 22:54:35.136827 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:35.136917 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":291,"threads_started":1}
I20260927 22:54:35.580194 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:35.579451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:40941)
0927 22:54:35.579967 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:35.580040 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":297,"threads_started":1}
I20260927 22:54:36.022668 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:36.021969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:43725)
0927 22:54:36.022422 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:36.022512 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":236,"threads_started":1}
I20260927 22:54:36.138572 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:36.137912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:45223)
0927 22:54:36.138306 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:36.138395 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":210,"threads_started":1}
I20260927 22:54:36.581779 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:36.581044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:45117)
0927 22:54:36.581473 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:36.581547 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":230,"threads_started":1}
I20260927 22:54:37.024330 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:37.023638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:53215)
0927 22:54:37.024092 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:37.024168 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":255,"threads_started":1}
I20260927 22:54:37.140116 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:37.139404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:51951)
0927 22:54:37.139792 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:37.139900 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":190,"threads_started":1}
I20260927 22:54:37.583308 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:37.582557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:46625)
0927 22:54:37.583056 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:37.583128 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":255,"threads_started":1}
I20260927 22:54:38.025988 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:38.025200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:38989)
0927 22:54:38.025742 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:38.025820 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":241,"threads_started":1}
I20260927 22:54:38.141691 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:38.140929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:60913)
0927 22:54:38.141395 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:38.141469 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":237,"threads_started":1}
I20260927 22:54:38.584826 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:38.584156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:43365)
0927 22:54:38.584597 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:38.584671 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":239,"threads_started":1}
I20260927 22:54:39.027573 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:39.026876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:60973)
0927 22:54:39.027325 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:39.027401 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":260,"threads_started":1}
I20260927 22:54:39.143168 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:39.142510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:44339)
0927 22:54:39.142911 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:39.143014 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":211,"threads_started":1}
I20260927 22:54:39.586675 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:39.585768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:56213)
0927 22:54:39.586260 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:39.586386 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260927 22:54:40.029208 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:40.028445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:53745)
0927 22:54:40.028922 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:40.029049 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":252,"threads_started":1}
I20260927 22:54:40.144752 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:40.144028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:45857)
0927 22:54:40.144467 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:40.144543 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":196,"threads_started":1}
I20260927 22:54:40.588447 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:40.587678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:35343)
0927 22:54:40.588163 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:40.588239 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":220,"threads_started":1}
I20260927 22:54:41.030876 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:41.030087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:32901)
0927 22:54:41.030575 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:41.030653 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":224,"threads_started":1}
I20260927 22:54:41.146407 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:41.145692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:54561)
0927 22:54:41.146150 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:41.146232 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260927 22:54:41.590013 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:41.589304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:49665)
0927 22:54:41.589782 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:41.589857 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":226,"threads_started":1}
I20260927 22:54:42.032423 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:42.031730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:34457)
0927 22:54:42.032193 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:42.032269 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
I20260927 22:54:42.147969 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:42.147244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:40355)
0927 22:54:42.147676 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:42.147756 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":215,"threads_started":1}
I20260927 22:54:42.591853 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:42.590916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:49627)
0927 22:54:42.591588 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:42.591673 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":430,"threads_started":1}
I20260927 22:54:43.034102 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:43.033276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:39169)
0927 22:54:43.033812 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:43.033926 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":244,"threads_started":1}
I20260927 22:54:43.149525 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:43.148854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:41529)
0927 22:54:43.149288 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:43.149362 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":246,"threads_started":1}
I20260927 22:54:43.593494 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:43.592798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:43451)
0927 22:54:43.593260 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:43.593339 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":258,"threads_started":1}
I20260927 22:54:44.035737 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:44.035047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:33217)
0927 22:54:44.035500 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:44.035582 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":242,"threads_started":1}
I20260927 22:54:44.151065 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:44.150401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:51643)
0927 22:54:44.150794 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:44.150889 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":194,"threads_started":1}
I20260927 22:54:44.602237 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:44.594461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:46863)
0927 22:54:44.601979 (+ 7518us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:44.602062 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7401,"thread_start_us":7284,"threads_started":1}
I20260927 22:54:45.037418 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:45.036713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:45657)
0927 22:54:45.037138 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:45.037223 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
I20260927 22:54:45.152477 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:45.151849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:44631)
0927 22:54:45.152246 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:45.152321 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":206,"threads_started":1}
I20260927 22:54:45.603844 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:45.603184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:56521)
0927 22:54:45.603605 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:45.603682 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":216,"threads_started":1}
I20260927 22:54:46.039191 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:46.038440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:34681)
0927 22:54:46.038866 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:46.038950 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":240,"threads_started":1}
I20260927 22:54:46.154104 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:46.153317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:47431)
0927 22:54:46.153790 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:46.153873 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":204,"threads_started":1}
I20260927 22:54:46.605512 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:46.604743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:38969)
0927 22:54:46.605232 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:46.605314 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":244,"threads_started":1}
I20260927 22:54:47.041109 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:47.040162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:42649)
0927 22:54:47.040791 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:47.040890 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":319,"threads_started":1}
I20260927 22:54:47.155604 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:47.154916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:53121)
0927 22:54:47.155352 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:47.155429 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
I20260927 22:54:47.607236 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:47.606455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:34213)
0927 22:54:47.606973 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:47.607054 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":233,"threads_started":1}
I20260927 22:54:48.042807 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:48.042047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:37401)
0927 22:54:48.042521 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:48.042605 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":237,"threads_started":1}
I20260927 22:54:48.157637 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:48.156712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.1:59039)
0927 22:54:48.157234 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:48.157332 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":276,"threads_started":1}
I20260927 22:54:48.608997 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:48.608292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.2:58913)
0927 22:54:48.608738 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:48.608822 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":238,"threads_started":1}
I20260927 22:54:49.019090 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 15381
I20260927 22:54:49.044725 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:49.043935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35627 (local address 127.0.97.3:38979)
0927 22:54:49.044445 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:49.044543 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":249,"threads_started":1}
I20260927 22:54:49.045187 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:49.020387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.0.97.1:42955)
0927 22:54:49.021101 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:49.021115 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:49.021164 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:54:49.022983 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:49.022992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:49.023460 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:49.023844 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:49.025262 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:49.026531 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:49.028292 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:49.029015 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:49.029217 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:49.038728 (+ 9511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:49.038735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:49.038751 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:49.038788 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:49.041685 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:49.042437 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:49.042451 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:49.042459 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:49.042625 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:49.043054 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:49.043061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:49.043065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:49.043630 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:49.043946 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:49.044342 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 22:54:49.044853 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":363,"threads_started":1}
I20260927 22:54:49.046696 15538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:49.131207 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 15381
I20260927 22:54:49.147691 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 15546
I20260927 22:54:49.164230 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:49.148649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.0.97.2:39339)
0927 22:54:49.149125 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:49.149138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:49.149175 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:54:49.150699 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:49.150708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:49.151134 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:49.151416 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:49.152767 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:49.154239 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:49.155446 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:49.156118 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:49.156245 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:49.158723 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:49.158730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:49.158744 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:49.158779 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:49.161455 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:49.162157 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:49.162163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:49.162170 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:49.162324 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:49.162632 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:49.162635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:49.162636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:49.163089 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:49.163310 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:49.163665 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 22:54:49.163963 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":215,"threads_started":1}
I20260927 22:54:49.165477 15669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:49.248325 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 15546
I20260927 22:54:49.264869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 15677
I20260927 22:54:49.281709 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:49.265791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.0.97.3:39533)
0927 22:54:49.266405 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:49.266419 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:49.266460 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:54:49.267991 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:49.268001 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:49.268424 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:49.268703 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:49.270119 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:49.271376 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:49.272635 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:49.273276 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:49.273420 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:49.275964 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:49.275972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:49.275985 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:49.276025 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:49.278809 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:49.279465 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:49.279473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:49.279479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:49.279630 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:49.279946 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:49.279948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:49.279950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:49.280449 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:49.280670 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:49.281050 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 22:54:49.281369 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":319,"threads_started":1}
I20260927 22:54:49.282998 15800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:49.363543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 15677
2026-09-27T22:54:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24362 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:54:49 dist-test-slave-df53 krb5kdc[15862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:54:49 dist-test-slave-df53 krb5kdc[15862](info): set up 2 sockets
Sep 27 22:54:49 dist-test-slave-df53 krb5kdc[15862](info): commencing operation
krb5kdc: starting...
W20260927 22:54:50.462100 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:54:50 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:54:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:54:50.681456 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35573
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:50.993806 15875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:50.994212 15875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:50.994393 15875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:51.001694 15875 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:54:51.001781 15875 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:54:51.001852 15875 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:54:51.001925 15875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:51.002007 15875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:51.002122 15875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:51.002194 15875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:51.002276 15875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:51.011893 15875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:51.013187 15875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:51.014556 15875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:51.026263 15880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:51.026212 15881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:51.026185 15883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:51.026857 15875 server_base.cc:1034] running on GCE node
I20260927 22:54:51.027642 15875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:51.028954 15875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:51.030191 15875 hybrid_clock.cc:648] HybridClock initialized: now 1790549691030128 us; error 37 us; skew 500 ppm
I20260927 22:54:51.030519 15875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:51 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:51.033545 15875 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:54:51.034981 15875 webserver.cc:460] Webserver started at http://127.0.97.62:39413/ using document root <none> and password file <none>
I20260927 22:54:51.035607 15875 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:51.035692 15875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:51.035938 15875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:51.038808 15875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e"
format_stamp: "Formatted at 2026-09-27 22:54:51 on dist-test-slave-df53"
server_key: "ef26f740aa00eb91067303e8a96f099f"
server_key_iv: "da045f74c09a4361c5644d4e3e0ee824"
server_key_version: "encryptionkey@0"
I20260927 22:54:51.039517 15875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e"
format_stamp: "Formatted at 2026-09-27 22:54:51 on dist-test-slave-df53"
server_key: "ef26f740aa00eb91067303e8a96f099f"
server_key_iv: "da045f74c09a4361c5644d4e3e0ee824"
server_key_version: "encryptionkey@0"
I20260927 22:54:51.044423 15875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:54:51.047694 15890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:51.049052 15875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:54:51.049233 15875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e"
format_stamp: "Formatted at 2026-09-27 22:54:51 on dist-test-slave-df53"
server_key: "ef26f740aa00eb91067303e8a96f099f"
server_key_iv: "da045f74c09a4361c5644d4e3e0ee824"
server_key_version: "encryptionkey@0"
I20260927 22:54:51.049454 15875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:51.069291 15875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:51.073356 15875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:51.073648 15875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:51.095083 15875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35573
I20260927 22:54:51.095096 15941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35573 every 8 connection(s)
I20260927 22:54:51.096375 15875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:51.099934 15942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:51.101097 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 15875
I20260927 22:54:51.101327 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:54:51.102245 388 external_mini_cluster.cc:1428] Setting key c50cdd6a802ac1bb2c5929c2834523b5
I20260927 22:54:51.115594 15942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Bootstrap starting.
Sep 27 22:54:51 dist-test-slave-df53 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:51.119127 15942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:51.120299 15942 log.cc:824] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:51.123288 15942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: No bootstrap required, opened a new log
I20260927 22:54:51.126351 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:51.104329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.0.97.62:35573)
0927 22:54:51.105201 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:51.105222 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:51.105274 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:54:51.107766 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:51.107794 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:51.108276 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:51.108615 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:51.110370 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:51.111626 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:51.113547 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:51.114251 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:51.114371 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:51.118969 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:51.118976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:51.118996 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:51.119037 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:51.122476 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:51.123127 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:51.123139 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:51.123144 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:51.123304 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:51.123716 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:51.123721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:51.123729 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:51.124200 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:51.124477 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:51.124931 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 22:54:51.125334 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":393,"threads_started":1}
I20260927 22:54:51.131618 15942 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:54:51.131963 15942 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:51.132066 15942 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ea0216fa7447f0ba7ba0fe21ba424e, State: Initialized, Role: FOLLOWER
I20260927 22:54:51.132616 15942 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:54:51.132890 15942 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:51.133057 15942 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:51.133262 15942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:51.135428 15942 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:54:51.135886 15942 leader_election.cc:302] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ea0216fa7447f0ba7ba0fe21ba424e; no voters:
I20260927 22:54:51.136425 15942 leader_election.cc:288] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:54:51.136579 15947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:51.137281 15947 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 1 LEADER]: Becoming Leader. State: Replica: b8ea0216fa7447f0ba7ba0fe21ba424e, State: Running, Role: LEADER
I20260927 22:54:51.137804 15947 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:54:51.138391 15942 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:51.140799 15949 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8ea0216fa7447f0ba7ba0fe21ba424e. Latest consensus state: current_term: 1 leader_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } } }
I20260927 22:54:51.141105 15948 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } } }
I20260927 22:54:51.141347 15949 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:51.141371 15948 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:51.143344 15953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:51.148662 15953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:51.157033 15953 catalog_manager.cc:1347] Generated new cluster ID: b359c426ee9247748a828d798c477718
I20260927 22:54:51.157137 15953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:51.166460 15953 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:54:51.167644 15953 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:51.175571 15953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Generated new TSK 0
I20260927 22:54:51.176327 15953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:54:51.231016 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35573
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:51.552635 15968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:51.552925 15968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:51.553035 15968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:51.560518 15968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:51.560617 15968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:51.560727 15968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:54:51.570459 15968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:51.572126 15968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:51.574040 15968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:51.589319 15973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:51.589373 15974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:51.589690 15976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:51.590325 15968 server_base.cc:1034] running on GCE node
I20260927 22:54:51.590965 15968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:51.591837 15968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:51.593079 15968 hybrid_clock.cc:648] HybridClock initialized: now 1790549691593014 us; error 59 us; skew 500 ppm
I20260927 22:54:51.593402 15968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:51 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:51.596637 15968 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:54:51.598174 15968 webserver.cc:460] Webserver started at http://127.0.97.1:45757/ using document root <none> and password file <none>
I20260927 22:54:51.598842 15968 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:51.598979 15968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:51.599318 15968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:51.602219 15968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "a2541e1068fe4d47907fb2d09336d68e"
format_stamp: "Formatted at 2026-09-27 22:54:51 on dist-test-slave-df53"
server_key: "6b8b726136ea3dd2b880a154f0413e06"
server_key_iv: "5605bc53ec35f6c6a8141883eae117b3"
server_key_version: "encryptionkey@0"
I20260927 22:54:51.602952 15968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a2541e1068fe4d47907fb2d09336d68e"
format_stamp: "Formatted at 2026-09-27 22:54:51 on dist-test-slave-df53"
server_key: "6b8b726136ea3dd2b880a154f0413e06"
server_key_iv: "5605bc53ec35f6c6a8141883eae117b3"
server_key_version: "encryptionkey@0"
I20260927 22:54:51.607954 15968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:54:51.611554 15983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:51.612799 15968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:54:51.612957 15968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "a2541e1068fe4d47907fb2d09336d68e"
format_stamp: "Formatted at 2026-09-27 22:54:51 on dist-test-slave-df53"
server_key: "6b8b726136ea3dd2b880a154f0413e06"
server_key_iv: "5605bc53ec35f6c6a8141883eae117b3"
server_key_version: "encryptionkey@0"
I20260927 22:54:51.613150 15968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:51.636564 15968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:51.640822 15968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:51.641110 15968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:51.642123 15968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:51.643371 15968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:51.643467 15968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:51.643610 15968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:51.643677 15968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:51.676155 15968 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43395
I20260927 22:54:51.676187 16095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43395 every 8 connection(s)
I20260927 22:54:51.677490 15968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:51.681770 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 15968
I20260927 22:54:51.681977 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:54:51.682585 388 external_mini_cluster.cc:1428] Setting key 41a1584b1cc017f892aa8b7eda6b142c
Sep 27 22:54:51 dist-test-slave-df53 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:51.696786 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:51.680019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42487 (local address 127.0.97.62:35573)
0927 22:54:51.680464 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:51.680468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:51.681331 (+ 863us) server_negotiation.cc:407] Connection header received
0927 22:54:51.683317 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:51.683322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:51.683426 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:51.683542 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:51.684996 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:51.685768 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:51.687281 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:51.687682 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:51.687816 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:51.691286 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:51.691291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:51.691294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:51.691325 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:51.693895 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:51.694583 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:51.694590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:51.694594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:51.694714 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:51.695115 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:51.695122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:51.695126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:51.695381 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:51.695536 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:51.696207 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 22:54:51.696427 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":235,"threads_started":1}
I20260927 22:54:51.697798 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:51.680489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:42487)
0927 22:54:51.681171 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:51.681216 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:51.683048 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:51.683779 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:51.683788 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:51.684281 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:51.684837 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:51.684853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:51.685913 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:51.685923 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:51.687137 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:51.687148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:51.687901 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:51.687910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:51.688147 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:51.688910 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:51.688932 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:51.691085 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:51.694023 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:51.694033 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:51.694049 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:51.694457 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:51.694819 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:51.694824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:51.694828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:51.695009 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:51.695511 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:51.695527 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:51.695847 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 22:54:51.696115 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 22:54:51.696641 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":258,"threads_started":1}
I20260927 22:54:51.699321 16096 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35573
I20260927 22:54:51.699616 16096 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:51.700289 16096 heartbeater.cc:511] Master 127.0.97.62:35573 requested a full tablet report, sending...
I20260927 22:54:51.702643 15907 ts_manager.cc:194] Registered new tserver with Master: a2541e1068fe4d47907fb2d09336d68e (127.0.97.1:43395)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:54:51.704720 15907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:54:51.732496 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35573
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:52.044106 16102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:52.044351 16102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:52.044440 16102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:52.051390 16102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:52.051483 16102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:52.051594 16102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:54:52.061092 16102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:52.062244 16102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:52.063426 16102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:52.075690 16107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:52.075762 16108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:52.075721 16110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:52.076820 16102 server_base.cc:1034] running on GCE node
I20260927 22:54:52.077497 16102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:52.078382 16102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:52.079564 16102 hybrid_clock.cc:648] HybridClock initialized: now 1790549692079503 us; error 53 us; skew 500 ppm
I20260927 22:54:52.079871 16102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:52 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:52.082894 16102 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:54:52.084277 16102 webserver.cc:460] Webserver started at http://127.0.97.2:36735/ using document root <none> and password file <none>
I20260927 22:54:52.084908 16102 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:52.085043 16102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:52.085345 16102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:52.088096 16102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a"
format_stamp: "Formatted at 2026-09-27 22:54:52 on dist-test-slave-df53"
server_key: "057d77b1ca92499dcc32c13ce0b181bc"
server_key_iv: "302afed457600533a2f2b4b2eaff57e8"
server_key_version: "encryptionkey@0"
I20260927 22:54:52.088810 16102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a"
format_stamp: "Formatted at 2026-09-27 22:54:52 on dist-test-slave-df53"
server_key: "057d77b1ca92499dcc32c13ce0b181bc"
server_key_iv: "302afed457600533a2f2b4b2eaff57e8"
server_key_version: "encryptionkey@0"
I20260927 22:54:52.093542 16102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 22:54:52.096828 16117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:52.098183 16102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:52.098338 16102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a"
format_stamp: "Formatted at 2026-09-27 22:54:52 on dist-test-slave-df53"
server_key: "057d77b1ca92499dcc32c13ce0b181bc"
server_key_iv: "302afed457600533a2f2b4b2eaff57e8"
server_key_version: "encryptionkey@0"
I20260927 22:54:52.098516 16102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:52.118966 16102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:52.122916 16102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:52.123207 16102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:52.124078 16102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:52.125254 16102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:52.125352 16102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:52.125485 16102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:52.125551 16102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:52.157779 16102 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34455
I20260927 22:54:52.157823 16229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34455 every 8 connection(s)
I20260927 22:54:52.159003 16102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:52.163046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 16102
I20260927 22:54:52.163247 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:54:52.163810 388 external_mini_cluster.cc:1428] Setting key 2f575d9be0b863b7e618eb16ca9bab96
Sep 27 22:54:52 dist-test-slave-df53 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:52.182197 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.161418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48225 (local address 127.0.97.62:35573)
0927 22:54:52.161616 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:52.161624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:52.162717 (+ 1093us) server_negotiation.cc:407] Connection header received
0927 22:54:52.165094 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:52.165098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:52.165166 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:52.165242 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:52.167198 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.167887 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.170290 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.170706 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.170797 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.175363 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:52.175372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:52.175380 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:52.175426 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:52.178977 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:52.179656 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:52.179664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:52.179671 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:52.179853 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:52.180258 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:52.180265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:52.180272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:52.180673 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:52.180822 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:52.181430 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 22:54:52.181759 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:54:52.182724 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.161886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:48225)
0927 22:54:52.162555 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:52.162591 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:52.164812 (+ 2221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:52.165495 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:52.165507 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:52.166245 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:52.167008 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.167048 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.168047 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.168055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.170158 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.170175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.170941 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.170951 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.171296 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.171977 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:52.172015 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:52.175093 (+ 3078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:52.179120 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:52.179132 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:52.179143 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:52.179530 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:52.179961 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:52.179964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:52.179967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:52.180139 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:52.180791 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:52.180796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:52.181075 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 22:54:52.181277 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 22:54:52.181686 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":270,"threads_started":1}
I20260927 22:54:52.184124 16230 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35573
I20260927 22:54:52.184433 16230 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:52.185259 16230 heartbeater.cc:511] Master 127.0.97.62:35573 requested a full tablet report, sending...
I20260927 22:54:52.186818 15907 ts_manager.cc:194] Registered new tserver with Master: a4c1b3b0a0e14a3da8573cc99e9d631a (127.0.97.2:34455)
I20260927 22:54:52.187602 15907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48225
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:54:52.217918 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35573
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:52.528249 16235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:52.528546 16235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:52.528710 16235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:52.535616 16235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:52.535738 16235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:52.535904 16235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:54:52.545332 16235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:54:52.546473 16235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:52.547678 16235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:52.560559 16240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:52.560530 16241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:52.560809 16243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:52.561682 16235 server_base.cc:1034] running on GCE node
I20260927 22:54:52.562268 16235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:52.563174 16235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:52.564361 16235 hybrid_clock.cc:648] HybridClock initialized: now 1790549692564303 us; error 53 us; skew 500 ppm
I20260927 22:54:52.564654 16235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:52 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:52.567638 16235 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:54:52.569084 16235 webserver.cc:460] Webserver started at http://127.0.97.3:45091/ using document root <none> and password file <none>
I20260927 22:54:52.569766 16235 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:52.569904 16235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:52.570212 16235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:52.572990 16235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "6fb52da8556e4e67aa61167f27e2b281"
format_stamp: "Formatted at 2026-09-27 22:54:52 on dist-test-slave-df53"
server_key: "239bfab5715e27888f10cd28bebab030"
server_key_iv: "ddde63222c2bccfb3aeecb50d4e876f5"
server_key_version: "encryptionkey@0"
I20260927 22:54:52.573805 16235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6fb52da8556e4e67aa61167f27e2b281"
format_stamp: "Formatted at 2026-09-27 22:54:52 on dist-test-slave-df53"
server_key: "239bfab5715e27888f10cd28bebab030"
server_key_iv: "ddde63222c2bccfb3aeecb50d4e876f5"
server_key_version: "encryptionkey@0"
I20260927 22:54:52.578684 16235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:54:52.582072 16250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:52.583307 16235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:54:52.583474 16235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "6fb52da8556e4e67aa61167f27e2b281"
format_stamp: "Formatted at 2026-09-27 22:54:52 on dist-test-slave-df53"
server_key: "239bfab5715e27888f10cd28bebab030"
server_key_iv: "ddde63222c2bccfb3aeecb50d4e876f5"
server_key_version: "encryptionkey@0"
I20260927 22:54:52.583675 16235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:52.617399 16235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:52.621161 16235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:52.621433 16235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:52.622362 16235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:52.623708 16235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:52.623803 16235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:52.623946 16235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:52.624014 16235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:52.657065 16235 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40415
I20260927 22:54:52.657140 16362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40415 every 8 connection(s)
I20260927 22:54:52.658281 16235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:52.667176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 16235
I20260927 22:54:52.667373 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:54:52.667889 388 external_mini_cluster.cc:1428] Setting key 09b1d09f5b740da2a53ae70294909a1a
Sep 27 22:54:52 dist-test-slave-df53 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:54:52.677850 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.660776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57319 (local address 127.0.97.62:35573)
0927 22:54:52.660982 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:52.660987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:52.662203 (+ 1216us) server_negotiation.cc:407] Connection header received
0927 22:54:52.663914 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:52.663918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:52.663985 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:52.664078 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:52.665424 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.666224 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.668224 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.668614 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.668706 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.672466 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:52.672471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:52.672474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:52.672506 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:52.675037 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:52.675733 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:52.675739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:52.675742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:52.675858 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:52.676260 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:52.676265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:52.676268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:52.676519 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:52.676670 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:52.677283 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 22:54:52.677489 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:54:52.678676 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.661259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:57319)
0927 22:54:52.662047 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:52.662091 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:52.663679 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:52.664281 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:52.664288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:52.664774 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:52.665285 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.665301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.666337 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.666340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.668090 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.668107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.668815 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.668824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.669089 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.669826 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:52.669853 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:52.672264 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:52.675192 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:52.675201 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:52.675214 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:52.675610 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:52.675960 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:52.675966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:52.675969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:52.676145 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:52.676640 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:52.676647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:52.676938 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 22:54:52.677186 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 22:54:52.677682 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":374,"threads_started":1}
I20260927 22:54:52.679913 16363 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35573
I20260927 22:54:52.680212 16363 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:52.680819 16363 heartbeater.cc:511] Master 127.0.97.62:35573 requested a full tablet report, sending...
I20260927 22:54:52.682256 15906 ts_manager.cc:194] Registered new tserver with Master: 6fb52da8556e4e67aa61167f27e2b281 (127.0.97.3:40415)
I20260927 22:54:52.683280 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:54:52.683413 15906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57319
I20260927 22:54:52.699805 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.687690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.0.97.62:35573)
0927 22:54:52.687867 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:52.687875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:52.688345 (+ 470us) server_negotiation.cc:407] Connection header received
0927 22:54:52.688478 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:52.688481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:52.688541 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:52.688617 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:52.690142 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.690816 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.692097 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.692473 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.692568 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.694854 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:52.694859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:52.694862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:52.694890 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:52.697405 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:52.698148 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:52.698152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:52.698154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:52.698269 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:52.698593 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:52.698596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:52.698598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:52.698871 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:52.699056 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:52.699348 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 22:54:52.699496 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:54:52.707849 16096 heartbeater.cc:503] Master 127.0.97.62:35573 was elected leader, sending a full tablet report...
I20260927 22:54:52.703835 15906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:54:52.711287 15906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:54:52.732776 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.723001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.0.97.2:34455)
0927 22:54:52.724279 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:52.724287 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:52.724317 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:54:52.724421 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:52.724428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:52.724692 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:52.725631 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:52.725847 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.728106 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.730080 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.731591 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.731717 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.731895 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:52.732299 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 22:54:52.732460 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":1659,"threads_started":1}
I20260927 22:54:52.732815 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.722800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34455 (local address 127.0.0.1:34116)
0927 22:54:52.723540 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:52.723582 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:52.723752 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:52.724978 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:52.724988 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:52.725055 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:52.725473 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.725487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.728277 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.728288 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.729924 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.729941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.731785 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.731791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.732073 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.732129 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:54:52.732247 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:54:52.732438 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":2,"thread_start_us":366,"threads_started":1}
I20260927 22:54:52.735116 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.725039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40415 (local address 127.0.0.1:41374)
0927 22:54:52.726230 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:52.726246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:52.726344 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:52.727801 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:52.727805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:52.727832 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:52.728365 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.728380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.730284 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.730289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.732461 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.732474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.734251 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.734257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.734431 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.734463 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:54:52.734578 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 22:54:52.734695 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":50,"thread_start_us":1020,"threads_started":1}
I20260927 22:54:52.735143 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.725794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.0.97.3:40415)
0927 22:54:52.727243 (+ 1449us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:52.727249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:52.727271 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:54:52.727362 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:52.727371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:52.727602 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:52.727812 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:52.728596 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.730135 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.732663 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.734062 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.734192 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.734407 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:52.734618 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 22:54:52.734766 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":1154,"threads_started":1}
I20260927 22:54:52.736464 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.722977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.0.1:37956)
0927 22:54:52.725281 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:52.725298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:52.725417 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:52.727644 (+ 2227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:52.727649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:52.727672 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:52.728130 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.728144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.729980 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.729987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.731954 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.731968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.735728 (+ 3760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.735734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.735880 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.735912 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:54:52.736021 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:54:52.736135 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2150,"mutex_wait_us":45,"thread_start_us":2683,"threads_started":1}
I20260927 22:54:52.736752 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.723640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.0.97.1:43395)
0927 22:54:52.726849 (+ 3209us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:52.726857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:52.726882 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:54:52.726976 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:52.726984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:52.727211 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:52.727423 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:52.728297 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.729836 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.732119 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.735275 (+ 3156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.735396 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.735598 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:52.736085 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 22:54:52.736242 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3050,"thread_start_us":2917,"threads_started":1}
I20260927 22:54:52.736925 16165 tablet_service.cc:1467] Processing CreateTablet for tablet d51f65e208eb43c2bdb356747317a3c8 (DEFAULT_TABLE table=test-table [id=a3e1604f415f461fb87cfd6557b93af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:52.738344 16165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d51f65e208eb43c2bdb356747317a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:52.738552 16298 tablet_service.cc:1467] Processing CreateTablet for tablet d51f65e208eb43c2bdb356747317a3c8 (DEFAULT_TABLE table=test-table [id=a3e1604f415f461fb87cfd6557b93af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:52.739872 16298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d51f65e208eb43c2bdb356747317a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:52.739989 16031 tablet_service.cc:1467] Processing CreateTablet for tablet d51f65e208eb43c2bdb356747317a3c8 (DEFAULT_TABLE table=test-table [id=a3e1604f415f461fb87cfd6557b93af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:52.741298 16031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d51f65e208eb43c2bdb356747317a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:52.749290 16381 tablet_bootstrap.cc:492] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: Bootstrap starting.
I20260927 22:54:52.752290 16381 tablet_bootstrap.cc:654] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:52.753647 16381 log.cc:824] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:52.754022 16382 tablet_bootstrap.cc:492] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: Bootstrap starting.
I20260927 22:54:52.757513 16384 tablet_bootstrap.cc:492] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: Bootstrap starting.
I20260927 22:54:52.759136 16382 tablet_bootstrap.cc:654] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:52.760294 16384 tablet_bootstrap.cc:654] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:52.760655 16382 log.cc:824] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:52.761519 16384 log.cc:824] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:52.762917 16381 tablet_bootstrap.cc:492] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: No bootstrap required, opened a new log
I20260927 22:54:52.763218 16381 ts_tablet_manager.cc:1397] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 22:54:52.763481 16382 tablet_bootstrap.cc:492] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: No bootstrap required, opened a new log
I20260927 22:54:52.763720 16382 ts_tablet_manager.cc:1397] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260927 22:54:52.765105 16384 tablet_bootstrap.cc:492] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: No bootstrap required, opened a new log
I20260927 22:54:52.765338 16384 ts_tablet_manager.cc:1397] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260927 22:54:52.770438 16382 raft_consensus.cc:348] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } }
I20260927 22:54:52.770773 16382 raft_consensus.cc:374] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:52.770888 16382 raft_consensus.cc:729] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb52da8556e4e67aa61167f27e2b281, State: Initialized, Role: FOLLOWER
I20260927 22:54:52.771715 16382 consensus_queue.cc:260] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } }
I20260927 22:54:52.773195 16363 heartbeater.cc:503] Master 127.0.97.62:35573 was elected leader, sending a full tablet report...
I20260927 22:54:52.773818 16382 ts_tablet_manager.cc:1428] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: Time spent starting tablet: real 0.010s user 0.009s sys 0.002s
I20260927 22:54:52.773681 16381 raft_consensus.cc:348] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } }
I20260927 22:54:52.774099 16381 raft_consensus.cc:374] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:52.774185 16381 raft_consensus.cc:729] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c1b3b0a0e14a3da8573cc99e9d631a, State: Initialized, Role: FOLLOWER
I20260927 22:54:52.774698 16381 consensus_queue.cc:260] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } }
I20260927 22:54:52.775763 16230 heartbeater.cc:503] Master 127.0.97.62:35573 was elected leader, sending a full tablet report...
I20260927 22:54:52.775630 16384 raft_consensus.cc:348] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } }
I20260927 22:54:52.776008 16384 raft_consensus.cc:374] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:52.776108 16384 raft_consensus.cc:729] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2541e1068fe4d47907fb2d09336d68e, State: Initialized, Role: FOLLOWER
I20260927 22:54:52.776366 16381 ts_tablet_manager.cc:1428] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260927 22:54:52.776717 16384 consensus_queue.cc:260] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } }
I20260927 22:54:52.778411 16384 ts_tablet_manager.cc:1428] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260927 22:54:52.873697 16387 raft_consensus.cc:482] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:54:52.874019 16387 raft_consensus.cc:504] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } }
I20260927 22:54:52.875731 16387 leader_election.cc:288] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4c1b3b0a0e14a3da8573cc99e9d631a (127.0.97.2:34455), a2541e1068fe4d47907fb2d09336d68e (127.0.97.1:43395)
I20260927 22:54:52.882081 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.876398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53967 (local address 127.0.97.1:43395)
0927 22:54:52.876596 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:52.876605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:52.876627 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:54:52.876704 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:52.876709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:52.876786 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:52.876890 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:52.877408 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.878353 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.880178 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.881214 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.881315 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.881385 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:52.881750 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 22:54:52.881839 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 22:54:52.882202 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.875991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.97.3:53967)
0927 22:54:52.876228 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:52.876248 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:52.876371 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:52.876913 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:52.876916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:52.876938 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:52.877266 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.877281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.878487 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.878492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.880009 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.880021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.881367 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.881373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.881563 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.881632 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 22:54:52.881731 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:54:52.881862 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"spinlock_wait_cycles":23936}
I20260927 22:54:52.882865 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.876091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34455 (local address 127.0.97.3:48623)
0927 22:54:52.876897 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:52.876913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:52.877026 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:52.877383 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:52.877387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:52.877412 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:52.877792 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.877804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.879097 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.879105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.880679 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:52.880692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.882075 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.882082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:52.882272 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.882309 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:54:52.882426 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 22:54:52.882564 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":938,"threads_started":1}
I20260927 22:54:52.882905 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.876237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48623 (local address 127.0.97.2:34455)
0927 22:54:52.876429 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:52.876438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:52.876995 (+ 557us) server_negotiation.cc:407] Connection header received
0927 22:54:52.877161 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:52.877166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:52.877245 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:52.877344 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:52.877945 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.878766 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.880821 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.881842 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.881964 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.882056 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:52.882553 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 22:54:52.882643 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:54:52.883247 16051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51f65e208eb43c2bdb356747317a3c8" candidate_uuid: "6fb52da8556e4e67aa61167f27e2b281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2541e1068fe4d47907fb2d09336d68e" is_pre_election: true
I20260927 22:54:52.883872 16051 raft_consensus.cc:2454] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb52da8556e4e67aa61167f27e2b281 in term 0.
I20260927 22:54:52.883914 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51f65e208eb43c2bdb356747317a3c8" candidate_uuid: "6fb52da8556e4e67aa61167f27e2b281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" is_pre_election: true
I20260927 22:54:52.884416 16185 raft_consensus.cc:2454] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb52da8556e4e67aa61167f27e2b281 in term 0.
I20260927 22:54:52.884519 16254 leader_election.cc:302] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb52da8556e4e67aa61167f27e2b281, a2541e1068fe4d47907fb2d09336d68e; no voters:
I20260927 22:54:52.884969 16387 raft_consensus.cc:2781] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:54:52.885102 16387 raft_consensus.cc:482] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:54:52.885236 16387 raft_consensus.cc:3037] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:52.887526 16387 raft_consensus.cc:504] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } }
I20260927 22:54:52.888108 16387 leader_election.cc:288] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [CANDIDATE]: Term 1 election: Requested vote from peers a4c1b3b0a0e14a3da8573cc99e9d631a (127.0.97.2:34455), a2541e1068fe4d47907fb2d09336d68e (127.0.97.1:43395)
I20260927 22:54:52.888548 16051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51f65e208eb43c2bdb356747317a3c8" candidate_uuid: "6fb52da8556e4e67aa61167f27e2b281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2541e1068fe4d47907fb2d09336d68e"
I20260927 22:54:52.888603 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51f65e208eb43c2bdb356747317a3c8" candidate_uuid: "6fb52da8556e4e67aa61167f27e2b281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a"
I20260927 22:54:52.888733 16051 raft_consensus.cc:3037] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:52.888856 16185 raft_consensus.cc:3037] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:52.890959 16051 raft_consensus.cc:2454] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fb52da8556e4e67aa61167f27e2b281 in term 1.
I20260927 22:54:52.891111 16185 raft_consensus.cc:2454] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fb52da8556e4e67aa61167f27e2b281 in term 1.
I20260927 22:54:52.891371 16254 leader_election.cc:302] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb52da8556e4e67aa61167f27e2b281, a2541e1068fe4d47907fb2d09336d68e; no voters:
I20260927 22:54:52.891675 16387 raft_consensus.cc:2781] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:52.892113 16387 raft_consensus.cc:686] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 1 LEADER]: Becoming Leader. State: Replica: 6fb52da8556e4e67aa61167f27e2b281, State: Running, Role: LEADER
I20260927 22:54:52.892606 16387 consensus_queue.cc:237] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } }
I20260927 22:54:52.896934 15905 catalog_manager.cc:5495] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fb52da8556e4e67aa61167f27e2b281 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6fb52da8556e4e67aa61167f27e2b281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } health_report { overall_health: HEALTHY } } }
W20260927 22:54:52.909854 16364 tablet.cc:2369] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:54:52.911026 16231 tablet.cc:2369] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:54:52.922452 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.917317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.0.97.3:40415)
0927 22:54:52.917534 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:52.917540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:52.917622 (+ 82us) server_negotiation.cc:407] Connection header received
0927 22:54:52.917828 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:52.917832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:52.917914 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:52.918049 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:54:52.918319 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.918997 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.920374 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.920761 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.920858 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.921323 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:54:52.921863 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:54:52.921969 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:52.922116 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 22:54:52.922177 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 22:54:52.929790 16097 tablet.cc:2369] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:54:52.930141 16185 raft_consensus.cc:1261] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 1 FOLLOWER]: Refusing update from remote peer 6fb52da8556e4e67aa61167f27e2b281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:52.930141 16051 raft_consensus.cc:1261] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e [term 1 FOLLOWER]: Refusing update from remote peer 6fb52da8556e4e67aa61167f27e2b281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:52.931052 16391 consensus_queue.cc:1035] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:52.931361 16387 consensus_queue.cc:1035] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:52.943079 16394 mvcc.cc:205] Tried to move back new op lower bound from 7334091542231855104 to 7334091542092410880. Current Snapshot: MvccSnapshot[applied={T|T < 7334091542231855104}]
I20260927 22:54:52.955087 15907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:54:52.955315 15907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:54:52.958621 15907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Sending DeleteTablet for 3 replicas of tablet d51f65e208eb43c2bdb356747317a3c8
I20260927 22:54:52.959578 16165 tablet_service.cc:1514] Processing DeleteTablet for tablet d51f65e208eb43c2bdb356747317a3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34116
I20260927 22:54:52.959895 16298 tablet_service.cc:1514] Processing DeleteTablet for tablet d51f65e208eb43c2bdb356747317a3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41374
I20260927 22:54:52.959887 16031 tablet_service.cc:1514] Processing DeleteTablet for tablet d51f65e208eb43c2bdb356747317a3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37956
I20260927 22:54:52.961114 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 15968
I20260927 22:54:52.970108 16405 tablet_replica.cc:330] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: stopping tablet replica
I20260927 22:54:52.970674 16405 raft_consensus.cc:2229] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:54:52.971297 16405 raft_consensus.cc:2258] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:52.973673 16405 ts_tablet_manager.cc:1905] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:52.978138 16404 tablet_replica.cc:330] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: stopping tablet replica
I20260927 22:54:52.978580 16404 raft_consensus.cc:2229] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:54:52.979084 16404 raft_consensus.cc:2258] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:52.981185 16404 ts_tablet_manager.cc:1905] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:52.984054 16405 ts_tablet_manager.cc:1918] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:52.984227 16405 log.cc:1196] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/d51f65e208eb43c2bdb356747317a3c8
I20260927 22:54:52.984845 16405 ts_tablet_manager.cc:1939] T d51f65e208eb43c2bdb356747317a3c8 P a2541e1068fe4d47907fb2d09336d68e: Deleting consensus metadata
I20260927 22:54:52.987020 15894 catalog_manager.cc:4841] TS a2541e1068fe4d47907fb2d09336d68e (127.0.97.1:43395): tablet d51f65e208eb43c2bdb356747317a3c8 (table test-table [id=a3e1604f415f461fb87cfd6557b93af2]) successfully deleted
I20260927 22:54:52.988348 16404 ts_tablet_manager.cc:1918] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:52.988507 16404 log.cc:1196] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/d51f65e208eb43c2bdb356747317a3c8
I20260927 22:54:52.989003 16404 ts_tablet_manager.cc:1939] T d51f65e208eb43c2bdb356747317a3c8 P a4c1b3b0a0e14a3da8573cc99e9d631a: Deleting consensus metadata
I20260927 22:54:52.990602 15893 catalog_manager.cc:4841] TS a4c1b3b0a0e14a3da8573cc99e9d631a (127.0.97.2:34455): tablet d51f65e208eb43c2bdb356747317a3c8 (table test-table [id=a3e1604f415f461fb87cfd6557b93af2]) successfully deleted
I20260927 22:54:52.992486 16406 tablet_replica.cc:330] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: stopping tablet replica
I20260927 22:54:52.992930 16406 raft_consensus.cc:2229] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:54:52.993559 16406 raft_consensus.cc:2258] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:52.995726 16406 ts_tablet_manager.cc:1905] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:54:52 dist-test-slave-df53 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:54:53.002538 16406 ts_tablet_manager.cc:1918] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:53.002703 16406 log.cc:1196] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/d51f65e208eb43c2bdb356747317a3c8
I20260927 22:54:53.003283 16406 ts_tablet_manager.cc:1939] T d51f65e208eb43c2bdb356747317a3c8 P 6fb52da8556e4e67aa61167f27e2b281: Deleting consensus metadata
I20260927 22:54:53.005518 15891 catalog_manager.cc:4841] TS 6fb52da8556e4e67aa61167f27e2b281 (127.0.97.3:40415): tablet d51f65e208eb43c2bdb356747317a3c8 (table test-table [id=a3e1604f415f461fb87cfd6557b93af2]) successfully deleted
I20260927 22:54:53.007045 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:52.962459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.0.97.1:43395)
0927 22:54:52.962682 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:52.962692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:52.986902 (+ 24210us) server_negotiation.cc:407] Connection header received
0927 22:54:52.987432 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:52.987436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:52.987503 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:52.987566 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:52.989308 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.990698 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.993054 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:52.993635 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:52.993752 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:52.999595 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:52.999610 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:52.999628 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:52.999671 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:53.003717 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:53.004600 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:53.004610 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:53.004615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:53.004776 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:53.005269 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:53.005273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:53.005276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:53.005708 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:53.005944 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:53.006465 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 22:54:53.006690 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:54:53.007632 16091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:53.102246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 15968
I20260927 22:54:53.125507 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 16102
Sep 27 22:54:53 dist-test-slave-df53 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:54:53.143289 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:53.126333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.0.97.2:34455)
0927 22:54:53.126532 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:53.126538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:53.126594 (+ 56us) server_negotiation.cc:407] Connection header received
0927 22:54:53.126860 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:53.126866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:53.126968 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:53.127082 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:53.128607 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:53.129909 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:53.131743 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:53.132361 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:53.132503 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:53.136820 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:53.136837 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:53.136855 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:53.136897 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:53.140580 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:53.141180 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:53.141190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:53.141194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:53.141334 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:53.141773 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:53.141778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:53.141782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:53.142225 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:53.142537 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:53.142704 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 22:54:53.142922 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:54:53.143975 16225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:53.235013 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 16102
I20260927 22:54:53.256494 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 16235
Sep 27 22:54:53 dist-test-slave-df53 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:54:53.280817 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:53.257572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.0.97.3:40415)
0927 22:54:53.258368 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:53.258378 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:53.258405 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:54:53.258493 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:53.258500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:53.258592 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:53.258717 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:53.261427 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:53.262749 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:53.265119 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:53.265843 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:53.265992 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:53.272444 (+ 6452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:53.272464 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:53.272487 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:53.272549 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:53.277097 (+ 4548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:53.278012 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:53.278023 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:53.278028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:53.278234 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:53.278692 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:53.278699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:53.278704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:53.279257 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:53.279587 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:53.280012 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 22:54:53.280286 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635}
I20260927 22:54:53.281807 16358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:53.406934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 16235
I20260927 22:54:53.426091 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 15875
I20260927 22:54:53.427115 15937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:53.534482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 15875
I20260927 22:55:13.555269 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35573
--webserver_interface=127.0.97.62
--webserver_port=39413
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:55:13.869112 16412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:13.869369 16412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:13.869454 16412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:13.876333 16412 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:55:13.876416 16412 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:55:13.876454 16412 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:55:13.876506 16412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:55:13.876559 16412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:13.876622 16412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:13.876667 16412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:55:13.876713 16412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:55:13.885950 16412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:13.887002 16412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:13.888226 16412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:13.910256 16419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:13.910295 16418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:13.910342 16421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:13.911449 16412 server_base.cc:1034] running on GCE node
I20260927 22:55:13.912338 16412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:13.913990 16412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:13.915287 16412 hybrid_clock.cc:648] HybridClock initialized: now 1790549713915208 us; error 47 us; skew 500 ppm
I20260927 22:55:13.915750 16412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:13 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:13.919433 16412 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:55:13.921191 16412 webserver.cc:460] Webserver started at http://127.0.97.62:39413/ using document root <none> and password file <none>
I20260927 22:55:13.922030 16412 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:13.922168 16412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:13.927877 16412 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:55:13.930734 16428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:13.932036 16412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:55:13.932220 16412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e"
format_stamp: "Formatted at 2026-09-27 22:54:51 on dist-test-slave-df53"
server_key: "ef26f740aa00eb91067303e8a96f099f"
server_key_iv: "da045f74c09a4361c5644d4e3e0ee824"
server_key_version: "encryptionkey@0"
I20260927 22:55:13.933213 16412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:13.953011 16412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:13.957217 16412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:13.957504 16412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:13.976193 16412 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35573
I20260927 22:55:13.976226 16479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35573 every 8 connection(s)
I20260927 22:55:13.977648 16412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:55:13.984475 16480 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:55:13.986579 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 16412
I20260927 22:55:13.987354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43395
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35573
--webserver_port=45757
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:55:13.988121 16480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Bootstrap starting.
I20260927 22:55:14.007402 16480 log.cc:824] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:14.018822 16480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:55:14.019352 16480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Bootstrap complete.
I20260927 22:55:14.028324 16480 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:55:14.029115 16480 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ea0216fa7447f0ba7ba0fe21ba424e, State: Initialized, Role: FOLLOWER
I20260927 22:55:14.029758 16480 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:55:14.029995 16480 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:55:14.030115 16480 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:55:14.030300 16480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:55:14.033962 16480 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:55:14.034435 16480 leader_election.cc:302] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ea0216fa7447f0ba7ba0fe21ba424e; no voters:
I20260927 22:55:14.035104 16480 leader_election.cc:288] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:55:14.035244 16484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:55:14.036681 16484 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 2 LEADER]: Becoming Leader. State: Replica: b8ea0216fa7447f0ba7ba0fe21ba424e, State: Running, Role: LEADER
I20260927 22:55:14.037403 16480 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:55:14.037377 16484 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:55:14.040015 16485 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } } }
I20260927 22:55:14.040416 16485 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: This master's current role is: LEADER
I20260927 22:55:14.041814 16486 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8ea0216fa7447f0ba7ba0fe21ba424e. Latest consensus state: current_term: 2 leader_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } } }
I20260927 22:55:14.042150 16486 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: This master's current role is: LEADER
I20260927 22:55:14.042290 16489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:55:14.049984 16489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:55:14.051326 16489 catalog_manager.cc:1259] Loaded cluster ID: b359c426ee9247748a828d798c477718
I20260927 22:55:14.051424 16489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:55:14.055132 16489 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:55:14.056592 16489 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Loaded TSK: 0
I20260927 22:55:14.087674 16489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Generated new TSK 1
I20260927 22:55:14.088006 16489 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:55:14.319373 16482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:14.319667 16482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:14.319826 16482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:14.326886 16482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:14.327039 16482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:14.327209 16482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:55:14.336695 16482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45757
--tserver_master_addrs=127.0.97.62:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:14.338038 16482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:14.339402 16482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:14.353636 16508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:14.353664 16507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:14.354532 16482 server_base.cc:1034] running on GCE node
W20260927 22:55:14.354773 16510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:14.355422 16482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:14.356302 16482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:14.357491 16482 hybrid_clock.cc:648] HybridClock initialized: now 1790549714357440 us; error 48 us; skew 500 ppm
I20260927 22:55:14.357802 16482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:14 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:14.360801 16482 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:55:14.362370 16482 webserver.cc:460] Webserver started at http://127.0.97.1:45757/ using document root <none> and password file <none>
I20260927 22:55:14.362993 16482 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:14.363128 16482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:14.368173 16482 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:55:14.370733 16517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:14.372103 16482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:55:14.372260 16482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "a2541e1068fe4d47907fb2d09336d68e"
format_stamp: "Formatted at 2026-09-27 22:54:51 on dist-test-slave-df53"
server_key: "6b8b726136ea3dd2b880a154f0413e06"
server_key_iv: "5605bc53ec35f6c6a8141883eae117b3"
server_key_version: "encryptionkey@0"
I20260927 22:55:14.373190 16482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:14.393558 16482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:14.397725 16482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:14.397995 16482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:14.399005 16482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:14.400424 16482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:14.400512 16482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:14.400673 16482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:14.400775 16482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:14.436834 16482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43395
I20260927 22:55:14.436918 16629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43395 every 8 connection(s)
I20260927 22:55:14.438548 16482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:55:14.448987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 16482
I20260927 22:55:14.449563 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34455
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35573
--webserver_port=36735
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:55:14 dist-test-slave-df53 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:55:14.461395 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:14.441538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:53373)
0927 22:55:14.442355 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:14.442397 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:14.444171 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:14.445691 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:14.445704 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:14.446304 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:14.446821 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:14.446838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.448422 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.448426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:14.449813 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:14.449829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.450844 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.450852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:14.451136 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:14.451796 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:55:14.451821 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:14.453925 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:14.457280 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:14.457288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:14.457301 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:14.457766 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:14.458197 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:14.458204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:14.458208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:14.458387 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:14.459089 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:14.459097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:14.459412 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 22:55:14.459658 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 22:55:14.460169 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":285,"threads_started":1}
I20260927 22:55:14.461395 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:14.441338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53373 (local address 127.0.97.62:35573)
0927 22:55:14.441971 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:14.441990 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:14.442528 (+ 538us) server_negotiation.cc:407] Connection header received
0927 22:55:14.444713 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:14.444727 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:14.445162 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:14.445509 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:14.447020 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.448249 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.449965 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.450594 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.450789 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:14.454194 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:14.454202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:14.454213 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:14.454247 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:14.457126 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:14.457911 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:14.457917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:14.457922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:14.458076 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:14.458509 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:14.458518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:14.458521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:14.458932 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:14.459137 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:14.459863 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 22:55:14.460228 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":282,"threads_started":1}
I20260927 22:55:14.464077 16630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35573
I20260927 22:55:14.464399 16630 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:14.465324 16630 heartbeater.cc:511] Master 127.0.97.62:35573 requested a full tablet report, sending...
I20260927 22:55:14.467574 16445 ts_manager.cc:194] Registered new tserver with Master: a2541e1068fe4d47907fb2d09336d68e (127.0.97.1:43395)
I20260927 22:55:14.469018 16445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53373
W20260927 22:55:14.761538 16634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:14.761806 16634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:14.761950 16634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:14.768949 16634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:14.769034 16634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:14.769259 16634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:55:14.778518 16634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36735
--tserver_master_addrs=127.0.97.62:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:14.779691 16634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:14.780890 16634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:14.793277 16639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:14.793220 16642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:14.793325 16640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:14.794243 16634 server_base.cc:1034] running on GCE node
I20260927 22:55:14.794812 16634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:14.795678 16634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:14.796893 16634 hybrid_clock.cc:648] HybridClock initialized: now 1790549714796848 us; error 41 us; skew 500 ppm
I20260927 22:55:14.797233 16634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:14 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:14.800336 16634 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:55:14.801698 16634 webserver.cc:460] Webserver started at http://127.0.97.2:36735/ using document root <none> and password file <none>
I20260927 22:55:14.802335 16634 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:14.802490 16634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:14.807379 16634 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:55:14.809993 16649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:14.811334 16634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:55:14.811528 16634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a"
format_stamp: "Formatted at 2026-09-27 22:54:52 on dist-test-slave-df53"
server_key: "057d77b1ca92499dcc32c13ce0b181bc"
server_key_iv: "302afed457600533a2f2b4b2eaff57e8"
server_key_version: "encryptionkey@0"
I20260927 22:55:14.812492 16634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:14.824966 16634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:14.829088 16634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:14.829360 16634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:14.830340 16634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:14.831583 16634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:14.831704 16634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:14.831828 16634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:14.831915 16634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:14.863991 16634 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34455
I20260927 22:55:14.864077 16761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34455 every 8 connection(s)
I20260927 22:55:14.865376 16634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:55:14.870505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 16634
I20260927 22:55:14.871099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40415
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35573
--webserver_port=45091
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:55:14 dist-test-slave-df53 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:55:14.887002 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:14.867924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34935 (local address 127.0.97.62:35573)
0927 22:55:14.868139 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:14.868143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:14.869253 (+ 1110us) server_negotiation.cc:407] Connection header received
0927 22:55:14.871455 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:14.871461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:14.871559 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:14.871672 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:14.873129 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.873894 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.875403 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.875828 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.875999 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:14.879916 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:14.879925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:14.879933 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:14.879980 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:14.883593 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:14.884322 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:14.884331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:14.884338 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:14.884486 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:14.884996 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:14.885004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:14.885010 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:14.885393 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:14.885548 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:14.886307 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 22:55:14.886568 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:55:14.887591 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:14.868343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:34935)
0927 22:55:14.869053 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:14.869101 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:14.871080 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:14.871913 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:14.871921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:14.872402 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:14.872947 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:14.872963 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.874042 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.874049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:14.875253 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:14.875264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.876104 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.876111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:14.876418 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:14.877184 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:55:14.877218 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:14.879670 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:14.883795 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:14.883801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:14.883812 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:14.884200 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:14.884700 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:14.884709 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:14.884711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:14.884875 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:14.885530 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:14.885535 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:14.885959 (+ 424us) client_negotiation.cc:769] Sending connection context
0927 22:55:14.886177 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 22:55:14.886555 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":300,"threads_started":1}
I20260927 22:55:14.889098 16762 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35573
I20260927 22:55:14.889402 16762 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:14.890151 16762 heartbeater.cc:511] Master 127.0.97.62:35573 requested a full tablet report, sending...
I20260927 22:55:14.891541 16445 ts_manager.cc:194] Registered new tserver with Master: a4c1b3b0a0e14a3da8573cc99e9d631a (127.0.97.2:34455)
I20260927 22:55:14.892324 16445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34935
W20260927 22:55:15.190979 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:15.191351 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:15.191586 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:15.199352 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:15.199488 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:15.199681 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:55:15.209102 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45091
--tserver_master_addrs=127.0.97.62:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:15.210335 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:15.211642 16765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:15.224313 16771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:15.224364 16773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:15.224401 16770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:15.225350 16765 server_base.cc:1034] running on GCE node
I20260927 22:55:15.225981 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:15.226876 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:15.228103 16765 hybrid_clock.cc:648] HybridClock initialized: now 1790549715228059 us; error 52 us; skew 500 ppm
I20260927 22:55:15.228410 16765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:15 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:15.231534 16765 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:55:15.232951 16765 webserver.cc:460] Webserver started at http://127.0.97.3:45091/ using document root <none> and password file <none>
I20260927 22:55:15.233672 16765 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:15.233827 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:15.239061 16765 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:55:15.241671 16780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:15.242964 16765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:55:15.243161 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "6fb52da8556e4e67aa61167f27e2b281"
format_stamp: "Formatted at 2026-09-27 22:54:52 on dist-test-slave-df53"
server_key: "239bfab5715e27888f10cd28bebab030"
server_key_iv: "ddde63222c2bccfb3aeecb50d4e876f5"
server_key_version: "encryptionkey@0"
I20260927 22:55:15.244251 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:15.267308 16765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:15.271376 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:15.271662 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:15.272650 16765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:15.274281 16765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:15.274439 16765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:15.274598 16765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:15.274698 16765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:15.310534 16765 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40415
I20260927 22:55:15.310796 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40415 every 8 connection(s)
I20260927 22:55:15.312098 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:55:15.315475 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 16765
Sep 27 22:55:15 dist-test-slave-df53 krb5kdc[15862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:55:15.334311 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.316825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.0.97.62:35573)
0927 22:55:15.317478 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.317484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.317835 (+ 351us) server_negotiation.cc:407] Connection header received
0927 22:55:15.318011 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.318015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.318086 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.318184 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.320482 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.321256 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.322698 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.323196 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.323335 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.326879 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.326886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.326891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.326929 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.330985 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.331653 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.331660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.331665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.331798 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.332267 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.332270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.332272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.332664 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.332916 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.333677 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.333909 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":342,"threads_started":1}
I20260927 22:55:15.336278 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.315224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:53447)
0927 22:55:15.316178 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.316235 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.318768 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.319527 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.319541 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.320280 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:15.320893 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.320908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.322029 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.322034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.323245 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.323257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.324034 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.324059 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.324380 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.324985 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:55:15.325025 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:15.327763 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:15.331709 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:15.331715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:15.331726 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:15.332321 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:15.332696 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:15.332703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:15.332710 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:15.332949 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:15.333651 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:15.333659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:15.334215 (+ 556us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.334535 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.334997 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":444,"threads_started":1}
I20260927 22:55:15.337352 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.314840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53447 (local address 127.0.97.62:35573)
0927 22:55:15.315045 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.315052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.316365 (+ 1313us) server_negotiation.cc:407] Connection header received
0927 22:55:15.319082 (+ 2717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.319086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.319134 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.319210 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.321077 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.321879 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.323387 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.323806 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.323953 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.327953 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.327959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.327964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.328013 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.331586 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.332419 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.332423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.332425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.332533 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.333093 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.333098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.333101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.333425 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.333576 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.336987 (+ 3411us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.337133 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:55:15.338392 16893 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35573
I20260927 22:55:15.338719 16893 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:15.339397 16893 heartbeater.cc:511] Master 127.0.97.62:35573 requested a full tablet report, sending...
I20260927 22:55:15.340728 16445 ts_manager.cc:194] Registered new tserver with Master: 6fb52da8556e4e67aa61167f27e2b281 (127.0.97.3:40415)
I20260927 22:55:15.341483 16445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53447
I20260927 22:55:15.349210 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:55:15.355309 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.350083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.0.97.62:35573)
0927 22:55:15.350248 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.350252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.350733 (+ 481us) server_negotiation.cc:407] Connection header received
0927 22:55:15.350893 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.350896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.351012 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.351145 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:55:15.351536 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.352251 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.353717 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.354119 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.354218 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.354694 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:55:15.354879 (+ 185us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 22:55:15.355124 (+ 245us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55422: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 22:55:15.355440 16632 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55422: token expired
W20260927 22:55:15.355942 16369 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:35573)
I20260927 22:55:15.367738 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.356364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.0.97.62:35573)
0927 22:55:15.356502 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.356506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.356581 (+ 75us) server_negotiation.cc:407] Connection header received
0927 22:55:15.356697 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.356700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.356749 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.356820 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.358186 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.358870 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.360111 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.360471 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.360560 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.362945 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.362950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.362953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.362979 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.365512 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.366132 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.366135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.366138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.366244 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.366571 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.366574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.366576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.366841 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.367104 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.367359 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.367501 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:55:15.371115 16445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:55:15.378633 16445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:55:15.403631 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.390048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.0.97.3:40415)
0927 22:55:15.395155 (+ 5107us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.395163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.395192 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:55:15.395310 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.395320 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.395573 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.396491 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.396777 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.398360 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.401004 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.402462 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.402608 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.402838 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.403129 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.403285 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4915,"thread_start_us":4786,"threads_started":1}
I20260927 22:55:15.403748 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.389764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40415 (local address 127.0.0.1:37650)
0927 22:55:15.390587 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.390626 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.390801 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.395823 (+ 5022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.395831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.395893 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.396278 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.396291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.398487 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.398491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.400811 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.400827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.402679 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.402685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.402883 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.402932 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.403069 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.403258 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":3,"spinlock_wait_cycles":22656,"thread_start_us":588,"threads_started":1}
I20260927 22:55:15.404479 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.391218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.0.97.2:34455)
0927 22:55:15.391866 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.391879 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.395781 (+ 3902us) server_negotiation.cc:407] Connection header received
0927 22:55:15.396061 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.396076 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.396252 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.396442 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.397456 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.398945 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.401512 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.402995 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.403156 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.403327 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.404004 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.404173 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":350,"threads_started":1}
I20260927 22:55:15.404481 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.390647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34455 (local address 127.0.0.1:36618)
0927 22:55:15.395674 (+ 5027us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.395694 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.395898 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.396439 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.396444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.396466 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.397135 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.397143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.399135 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.399146 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.401327 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.401342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.403505 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.403511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.403742 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.403812 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.403934 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.404060 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4858,"mutex_wait_us":95,"spinlock_wait_cycles":1792,"thread_start_us":4668,"threads_started":1}
I20260927 22:55:15.404652 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.390540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.0.97.1:43395)
0927 22:55:15.395939 (+ 5399us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.395948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.395971 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:55:15.396067 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.396076 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.396313 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.396710 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.398550 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.399680 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.401868 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.403087 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.403211 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.403390 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.404183 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.404338 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5235,"thread_start_us":5105,"threads_started":1}
I20260927 22:55:15.404948 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.390160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.0.1:60398)
0927 22:55:15.394392 (+ 4232us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.394417 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.394540 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.396508 (+ 1968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.396512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.396535 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.397127 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.397140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.399814 (+ 2674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.399820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.401711 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.401727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.403557 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.403563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.403797 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.403835 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.403923 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.404051 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4081,"mutex_wait_us":25,"thread_start_us":3949,"threads_started":1}
I20260927 22:55:15.408003 16565 tablet_service.cc:1467] Processing CreateTablet for tablet 9a159cd9ad4849e7a9bdb56f8b17acb1 (DEFAULT_TABLE table=test-table [id=e7247d15fc0e445b89eb087cbdccb3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:15.408174 16828 tablet_service.cc:1467] Processing CreateTablet for tablet 9a159cd9ad4849e7a9bdb56f8b17acb1 (DEFAULT_TABLE table=test-table [id=e7247d15fc0e445b89eb087cbdccb3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:15.409271 16565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a159cd9ad4849e7a9bdb56f8b17acb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:15.408914 16697 tablet_service.cc:1467] Processing CreateTablet for tablet 9a159cd9ad4849e7a9bdb56f8b17acb1 (DEFAULT_TABLE table=test-table [id=e7247d15fc0e445b89eb087cbdccb3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:15.409499 16828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a159cd9ad4849e7a9bdb56f8b17acb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:15.410104 16697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a159cd9ad4849e7a9bdb56f8b17acb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:15.421836 16909 tablet_bootstrap.cc:492] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: Bootstrap starting.
I20260927 22:55:15.424762 16910 tablet_bootstrap.cc:492] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: Bootstrap starting.
I20260927 22:55:15.426445 16911 tablet_bootstrap.cc:492] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: Bootstrap starting.
I20260927 22:55:15.427641 16909 tablet_bootstrap.cc:654] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:15.428189 16910 tablet_bootstrap.cc:654] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:15.428923 16909 log.cc:824] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:15.429158 16911 tablet_bootstrap.cc:654] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:15.429370 16910 log.cc:824] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:15.430357 16911 log.cc:824] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:15.432017 16909 tablet_bootstrap.cc:492] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: No bootstrap required, opened a new log
I20260927 22:55:15.432227 16910 tablet_bootstrap.cc:492] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: No bootstrap required, opened a new log
I20260927 22:55:15.432343 16909 ts_tablet_manager.cc:1397] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260927 22:55:15.432474 16910 ts_tablet_manager.cc:1397] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:55:15.433218 16911 tablet_bootstrap.cc:492] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: No bootstrap required, opened a new log
I20260927 22:55:15.433449 16911 ts_tablet_manager.cc:1397] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 22:55:15.442039 16909 raft_consensus.cc:348] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } }
I20260927 22:55:15.442474 16909 raft_consensus.cc:374] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:15.442569 16909 raft_consensus.cc:729] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2541e1068fe4d47907fb2d09336d68e, State: Initialized, Role: FOLLOWER
I20260927 22:55:15.443218 16909 consensus_queue.cc:260] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } }
I20260927 22:55:15.443427 16910 raft_consensus.cc:348] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } }
I20260927 22:55:15.443796 16910 raft_consensus.cc:374] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:15.443895 16910 raft_consensus.cc:729] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c1b3b0a0e14a3da8573cc99e9d631a, State: Initialized, Role: FOLLOWER
I20260927 22:55:15.444199 16911 raft_consensus.cc:348] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } }
I20260927 22:55:15.444581 16911 raft_consensus.cc:374] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:15.444681 16911 raft_consensus.cc:729] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb52da8556e4e67aa61167f27e2b281, State: Initialized, Role: FOLLOWER
I20260927 22:55:15.444490 16910 consensus_queue.cc:260] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } }
I20260927 22:55:15.445399 16911 consensus_queue.cc:260] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } }
I20260927 22:55:15.449570 16911 ts_tablet_manager.cc:1428] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260927 22:55:15.450326 16893 heartbeater.cc:503] Master 127.0.97.62:35573 was elected leader, sending a full tablet report...
I20260927 22:55:15.458242 16630 heartbeater.cc:503] Master 127.0.97.62:35573 was elected leader, sending a full tablet report...
I20260927 22:55:15.459167 16910 ts_tablet_manager.cc:1428] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: Time spent starting tablet: real 0.027s user 0.024s sys 0.004s
I20260927 22:55:15.459399 16762 heartbeater.cc:503] Master 127.0.97.62:35573 was elected leader, sending a full tablet report...
I20260927 22:55:15.461050 16909 ts_tablet_manager.cc:1428] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: Time spent starting tablet: real 0.029s user 0.022s sys 0.005s
I20260927 22:55:15.479398 16916 raft_consensus.cc:482] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:55:15.479651 16916 raft_consensus.cc:504] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } }
I20260927 22:55:15.481281 16916 leader_election.cc:288] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fb52da8556e4e67aa61167f27e2b281 (127.0.97.3:40415), a2541e1068fe4d47907fb2d09336d68e (127.0.97.1:43395)
I20260927 22:55:15.484323 16915 raft_consensus.cc:482] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:55:15.484601 16915 raft_consensus.cc:504] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } }
I20260927 22:55:15.490268 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.481756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50353 (local address 127.0.97.1:43395)
0927 22:55:15.481951 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.481959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.482229 (+ 270us) server_negotiation.cc:407] Connection header received
0927 22:55:15.482498 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.482503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.482586 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.482692 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.483450 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.484528 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.487209 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.488511 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.488632 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.488738 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.489831 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.489971 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:55:15.490550 16915 leader_election.cc:288] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fb52da8556e4e67aa61167f27e2b281 (127.0.97.3:40415), a4c1b3b0a0e14a3da8573cc99e9d631a (127.0.97.2:34455)
I20260927 22:55:15.491101 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.481513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.97.2:50353)
0927 22:55:15.482090 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.482114 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.482286 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.482724 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.482727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.482751 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.483306 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.483320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.484658 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.484666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.487006 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.487024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.488988 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.488998 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.489207 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.489244 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.490520 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.490678 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"spinlock_wait_cycles":30592,"thread_start_us":900,"threads_started":1}
I20260927 22:55:15.491714 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.481845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50287 (local address 127.0.97.3:40415)
0927 22:55:15.482034 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.482038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.482883 (+ 845us) server_negotiation.cc:407] Connection header received
0927 22:55:15.483122 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.483137 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.483247 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.483354 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.483959 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.484952 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.487374 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.488615 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.488695 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.488788 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.491340 (+ 2552us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.491462 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:55:15.491797 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.481529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40415 (local address 127.0.97.2:50287)
0927 22:55:15.482758 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.482783 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.482977 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.483392 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.483398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.483425 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.483794 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.483808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.485077 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.485082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.487213 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.487225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.489395 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.489404 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.489614 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.489668 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.491318 (+ 1650us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.491446 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"thread_start_us":447,"threads_started":1}
I20260927 22:55:15.492055 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a159cd9ad4849e7a9bdb56f8b17acb1" candidate_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2541e1068fe4d47907fb2d09336d68e" is_pre_election: true
I20260927 22:55:15.492543 16585 raft_consensus.cc:2454] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4c1b3b0a0e14a3da8573cc99e9d631a in term 0.
I20260927 22:55:15.493003 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a159cd9ad4849e7a9bdb56f8b17acb1" candidate_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb52da8556e4e67aa61167f27e2b281" is_pre_election: true
I20260927 22:55:15.493616 16848 raft_consensus.cc:2454] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4c1b3b0a0e14a3da8573cc99e9d631a in term 0.
I20260927 22:55:15.493901 16653 leader_election.cc:302] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2541e1068fe4d47907fb2d09336d68e, a4c1b3b0a0e14a3da8573cc99e9d631a; no voters:
I20260927 22:55:15.494361 16916 raft_consensus.cc:2781] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:55:15.494505 16916 raft_consensus.cc:482] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:55:15.494642 16916 raft_consensus.cc:3037] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:15.497424 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.487712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48045 (local address 127.0.97.2:34455)
0927 22:55:15.487886 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.487894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.489521 (+ 1627us) server_negotiation.cc:407] Connection header received
0927 22:55:15.489726 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.489734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.489826 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.489910 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.491235 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.492254 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.495216 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.496445 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.496621 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.496713 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.497048 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.497185 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 22:55:15.497471 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.486727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34455 (local address 127.0.97.1:48045)
0927 22:55:15.489436 (+ 2709us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.489458 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.489575 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.490773 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.490777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.490798 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.491070 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.491080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.492412 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.492417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.495059 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.495071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.496688 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.496694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.496853 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.496910 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.497011 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.497145 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2551,"thread_start_us":3534,"threads_started":1}
I20260927 22:55:15.498291 16916 raft_consensus.cc:504] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } }
I20260927 22:55:15.498418 16717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a159cd9ad4849e7a9bdb56f8b17acb1" candidate_uuid: "a2541e1068fe4d47907fb2d09336d68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" is_pre_election: true
I20260927 22:55:15.498874 16717 raft_consensus.cc:2379] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2541e1068fe4d47907fb2d09336d68e in current term 1: Already voted for candidate a4c1b3b0a0e14a3da8573cc99e9d631a in this term.
I20260927 22:55:15.499377 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.486727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40415 (local address 127.0.97.1:34893)
0927 22:55:15.490595 (+ 3868us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.490611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.490720 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.491094 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.491097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.491122 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.492743 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.492758 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.494720 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.494726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.496782 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.496798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.498543 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.498548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.498727 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.498756 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.498883 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.499010 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3739,"mutex_wait_us":552,"thread_start_us":1760,"threads_started":1}
I20260927 22:55:15.499418 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a159cd9ad4849e7a9bdb56f8b17acb1" candidate_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb52da8556e4e67aa61167f27e2b281"
I20260927 22:55:15.499655 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.487004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34893 (local address 127.0.97.3:40415)
0927 22:55:15.490104 (+ 3100us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.490110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.490820 (+ 710us) server_negotiation.cc:407] Connection header received
0927 22:55:15.490878 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.490882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.490947 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.491063 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.493007 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.494439 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.496929 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.498329 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.498478 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.498802 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.498901 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.499074 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2973,"thread_start_us":2780,"threads_started":1}
I20260927 22:55:15.499650 16848 raft_consensus.cc:3037] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:15.499931 16916 leader_election.cc:288] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [CANDIDATE]: Term 1 election: Requested vote from peers 6fb52da8556e4e67aa61167f27e2b281 (127.0.97.3:40415), a2541e1068fe4d47907fb2d09336d68e (127.0.97.1:43395)
I20260927 22:55:15.500375 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a159cd9ad4849e7a9bdb56f8b17acb1" candidate_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2541e1068fe4d47907fb2d09336d68e"
I20260927 22:55:15.500574 16585 raft_consensus.cc:3037] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:15.501190 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a159cd9ad4849e7a9bdb56f8b17acb1" candidate_uuid: "a2541e1068fe4d47907fb2d09336d68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb52da8556e4e67aa61167f27e2b281" is_pre_election: true
I20260927 22:55:15.502175 16518 leader_election.cc:302] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2541e1068fe4d47907fb2d09336d68e; no voters: 6fb52da8556e4e67aa61167f27e2b281, a4c1b3b0a0e14a3da8573cc99e9d631a
I20260927 22:55:15.502728 16848 raft_consensus.cc:2454] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4c1b3b0a0e14a3da8573cc99e9d631a in term 1.
I20260927 22:55:15.503077 16650 leader_election.cc:302] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb52da8556e4e67aa61167f27e2b281, a4c1b3b0a0e14a3da8573cc99e9d631a; no voters:
I20260927 22:55:15.503376 16916 raft_consensus.cc:2781] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:55:15.503777 16585 raft_consensus.cc:2454] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4c1b3b0a0e14a3da8573cc99e9d631a in term 1.
I20260927 22:55:15.503839 16916 raft_consensus.cc:686] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 1 LEADER]: Becoming Leader. State: Replica: a4c1b3b0a0e14a3da8573cc99e9d631a, State: Running, Role: LEADER
I20260927 22:55:15.504031 16915 raft_consensus.cc:2726] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:55:15.504390 16916 consensus_queue.cc:237] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } }
I20260927 22:55:15.508769 16445 catalog_manager.cc:5495] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a reported cstate change: term changed from 0 to 1, leader changed from <none> to a4c1b3b0a0e14a3da8573cc99e9d631a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34455 } health_report { overall_health: HEALTHY } } }
I20260927 22:55:15.531397 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.526403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.0.97.2:34455)
0927 22:55:15.526634 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.526639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.526720 (+ 81us) server_negotiation.cc:407] Connection header received
0927 22:55:15.526886 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.526890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.526976 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.527093 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:55:15.527398 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.528096 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.529435 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.529849 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.529942 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.530388 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:55:15.530849 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:55:15.530950 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.531017 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.531086 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 22:55:15.539165 16848 raft_consensus.cc:1261] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281 [term 1 FOLLOWER]: Refusing update from remote peer a4c1b3b0a0e14a3da8573cc99e9d631a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:55:15.539252 16585 raft_consensus.cc:1261] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 1 FOLLOWER]: Refusing update from remote peer a4c1b3b0a0e14a3da8573cc99e9d631a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:55:15.540014 16916 consensus_queue.cc:1035] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb52da8556e4e67aa61167f27e2b281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:55:15.540359 16923 consensus_queue.cc:1035] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2541e1068fe4d47907fb2d09336d68e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:55:15.565145 16444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:55:15.565409 16444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:55:15.569118 16444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Sending DeleteTablet for 3 replicas of tablet 9a159cd9ad4849e7a9bdb56f8b17acb1
I20260927 22:55:15.570477 16828 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a159cd9ad4849e7a9bdb56f8b17acb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37650
I20260927 22:55:15.570454 16565 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a159cd9ad4849e7a9bdb56f8b17acb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60398
I20260927 22:55:15.570796 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 16482
I20260927 22:55:15.576468 16697 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a159cd9ad4849e7a9bdb56f8b17acb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36618
I20260927 22:55:15.584735 16938 tablet_replica.cc:330] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: stopping tablet replica
I20260927 22:55:15.585259 16938 raft_consensus.cc:2229] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:55:15.586433 16936 tablet_replica.cc:330] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: stopping tablet replica
I20260927 22:55:15.586757 16937 tablet_replica.cc:330] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: stopping tablet replica
I20260927 22:55:15.586862 16936 raft_consensus.cc:2229] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:55:15.587175 16937 raft_consensus.cc:2229] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:55:15.587352 16936 raft_consensus.cc:2258] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:15.585937 16938 raft_consensus.cc:2258] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:15.587683 16937 raft_consensus.cc:2258] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:15.589494 16938 ts_tablet_manager.cc:1905] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:15.589759 16937 ts_tablet_manager.cc:1905] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:15.593276 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.571997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.0.97.1:43395)
0927 22:55:15.572174 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.572183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.572205 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:55:15.572268 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.572275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.572372 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.572488 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.574966 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.576168 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.580799 (+ 4631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.581701 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.581863 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.585662 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.585683 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.585707 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.585766 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.589990 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.590882 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.590892 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.590895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.591077 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.591489 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.591494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.591497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.591993 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.592271 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.592597 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.592808 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:55:15.593993 16625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:15.596971 16936 ts_tablet_manager.cc:1905] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:15.599670 16938 ts_tablet_manager.cc:1918] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:15.605494 16936 ts_tablet_manager.cc:1918] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:15.605686 16936 log.cc:1196] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/9a159cd9ad4849e7a9bdb56f8b17acb1
I20260927 22:55:15.605855 16938 log.cc:1196] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/9a159cd9ad4849e7a9bdb56f8b17acb1
I20260927 22:55:15.606225 16936 ts_tablet_manager.cc:1939] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P 6fb52da8556e4e67aa61167f27e2b281: Deleting consensus metadata
I20260927 22:55:15.606616 16938 ts_tablet_manager.cc:1939] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a4c1b3b0a0e14a3da8573cc99e9d631a: Deleting consensus metadata
I20260927 22:55:15.608069 16429 catalog_manager.cc:4841] TS 6fb52da8556e4e67aa61167f27e2b281 (127.0.97.3:40415): tablet 9a159cd9ad4849e7a9bdb56f8b17acb1 (table test-table [id=e7247d15fc0e445b89eb087cbdccb3d0]) successfully deleted
I20260927 22:55:15.608487 16431 catalog_manager.cc:4841] TS a4c1b3b0a0e14a3da8573cc99e9d631a (127.0.97.2:34455): tablet 9a159cd9ad4849e7a9bdb56f8b17acb1 (table test-table [id=e7247d15fc0e445b89eb087cbdccb3d0]) successfully deleted
I20260927 22:55:15.689029 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 16482
I20260927 22:55:15.707039 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 16634
W20260927 22:55:15.707096 16432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43395 (error 108)
I20260927 22:55:15.709273 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.708903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.0.1:60416)
0927 22:55:15.709045 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.709136 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 22:55:15.709582 16432 catalog_manager.cc:4584] TS a2541e1068fe4d47907fb2d09336d68e (127.0.97.1:43395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a159cd9ad4849e7a9bdb56f8b17acb1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43395: connect: Connection refused (error 111)
I20260927 22:55:15.726008 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.707983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.0.97.2:34455)
0927 22:55:15.708291 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.708298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.708322 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:55:15.708539 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.708545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.708630 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.708748 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.710551 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.711339 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.712759 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.713216 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.713355 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.719801 (+ 6446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.719812 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.719826 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.719867 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.723165 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.723914 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.723921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.723923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.724071 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.724450 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.724456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.724459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.724822 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.725112 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.725431 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.725690 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260927 22:55:15.726822 16757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:15.763242 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.762698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.0.1:60428)
0927 22:55:15.762925 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.763041 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:55:15.764369 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.763964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.0.1:60444)
0927 22:55:15.764131 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.764226 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:55:15.824831 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.824264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.0.1:60456)
0927 22:55:15.824524 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.824632 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 22:55:15.825876 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.825497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.0.1:60460)
0927 22:55:15.825671 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.825753 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 22:55:15.839079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 16634
I20260927 22:55:15.856961 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 16765
I20260927 22:55:15.870046 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.857665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.0.97.3:40415)
0927 22:55:15.857840 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.857846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.857888 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:55:15.858140 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.858144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.858218 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.858327 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.859628 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.860334 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.861564 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.862058 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.862197 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.864652 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.864666 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.864681 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.864714 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.867256 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.867951 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.867958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.867960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.868081 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.868485 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.868494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.868496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.868835 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.869136 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.869440 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.869707 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:55:15.870734 16888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:15.906739 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.906198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.0.1:60462)
0927 22:55:15.906435 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.906551 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:55:15.908082 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.907607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.0.1:60478)
0927 22:55:15.907838 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.907937 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 22:55:15.960721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 16765
I20260927 22:55:15.977062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 16412
I20260927 22:55:15.978338 16475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:16.091941 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:16.091345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43395 (local address 127.0.0.1:60484)
0927 22:55:16.091564 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:16.091677 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 22:55:16.092233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 16412
I20260927 22:55:36.110335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35573
--webserver_interface=127.0.97.62
--webserver_port=39413
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:55:36.431411 16944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:36.431710 16944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:36.431900 16944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:36.439016 16944 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:55:36.439102 16944 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:55:36.439203 16944 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:55:36.439296 16944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:55:36.439393 16944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:36.439510 16944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:36.439594 16944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:55:36.439677 16944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:55:36.449133 16944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:36.450358 16944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:36.451643 16944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:36.462869 16952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:36.462927 16949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:36.462949 16950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:36.463789 16944 server_base.cc:1034] running on GCE node
I20260927 22:55:36.464533 16944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:36.465858 16944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:36.467104 16944 hybrid_clock.cc:648] HybridClock initialized: now 1790549736467054 us; error 36 us; skew 500 ppm
I20260927 22:55:36.467440 16944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:36 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:36.470557 16944 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:55:36.471930 16944 webserver.cc:460] Webserver started at http://127.0.97.62:39413/ using document root <none> and password file <none>
I20260927 22:55:36.472590 16944 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:36.472718 16944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:36.477905 16944 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:55:36.480494 16959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:36.481761 16944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:55:36.481931 16944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e"
format_stamp: "Formatted at 2026-09-27 22:54:51 on dist-test-slave-df53"
server_key: "ef26f740aa00eb91067303e8a96f099f"
server_key_iv: "da045f74c09a4361c5644d4e3e0ee824"
server_key_version: "encryptionkey@0"
I20260927 22:55:36.482854 16944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:36.504565 16944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:36.508729 16944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:36.509003 16944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:36.527542 16944 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35573
I20260927 22:55:36.527619 17010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35573 every 8 connection(s)
I20260927 22:55:36.528939 16944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:55:36.531474 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 16944
I20260927 22:55:36.532270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43395
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35573
--webserver_port=45757
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:55:36.535564 17011 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:55:36.538959 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Bootstrap starting.
I20260927 22:55:36.565328 17011 log.cc:824] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:36.580960 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:55:36.581496 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Bootstrap complete.
I20260927 22:55:36.590986 17011 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:55:36.591903 17011 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ea0216fa7447f0ba7ba0fe21ba424e, State: Initialized, Role: FOLLOWER
I20260927 22:55:36.592523 17011 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:55:36.592751 17011 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:55:36.592869 17011 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:55:36.593061 17011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:55:36.596714 17011 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:55:36.597191 17011 leader_election.cc:302] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ea0216fa7447f0ba7ba0fe21ba424e; no voters:
I20260927 22:55:36.597869 17011 leader_election.cc:288] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:55:36.597997 17015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:55:36.599185 17015 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [term 3 LEADER]: Becoming Leader. State: Replica: b8ea0216fa7447f0ba7ba0fe21ba424e, State: Running, Role: LEADER
I20260927 22:55:36.599756 17015 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } }
I20260927 22:55:36.600028 17011 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:55:36.602209 17017 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8ea0216fa7447f0ba7ba0fe21ba424e. Latest consensus state: current_term: 3 leader_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } } }
I20260927 22:55:36.602596 17017 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: This master's current role is: LEADER
I20260927 22:55:36.603161 17016 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ea0216fa7447f0ba7ba0fe21ba424e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35573 } } }
I20260927 22:55:36.603469 17016 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e [sys.catalog]: This master's current role is: LEADER
I20260927 22:55:36.614591 17021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:55:36.623715 17021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:55:36.625195 17021 catalog_manager.cc:1259] Loaded cluster ID: b359c426ee9247748a828d798c477718
I20260927 22:55:36.625355 17021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:55:36.629288 17021 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:55:36.630975 17021 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Loaded TSK: 0
I20260927 22:55:36.631235 17021 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b8ea0216fa7447f0ba7ba0fe21ba424e: Loaded TSK: 1
F20260927 22:55:36.632483 17021 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6a7b7ec9ad google::LogMessage::Fail() at ??:0
@ 0x7f6a7b7ef88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6a7b7ec4de google::LogMessage::Flush() at ??:0
@ 0x7f6a7b7eda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6a7f5507d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6a7f53f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6a9121b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6a9121a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6a912a62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f6a912a60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f6a912f05e9 std::function<>::operator()() at ??:0
@ 0x7f6a9121ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6a9121cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6a912a4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6a912a4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6a7e438345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6a7e43b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6a7e43b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6a7e416eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6a816f06db start_thread at ??:0
@ 0x7f6a7a68661f clone at ??:0
W20260927 22:55:36.926453 17013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:36.926739 17013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:36.926880 17013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:36.933892 17013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:36.933988 17013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:36.934098 17013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:55:36.943652 17013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45757
--tserver_master_addrs=127.0.97.62:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:36.944824 17013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:36.946072 17013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:36.958545 17058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:36.958629 17061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:36.958657 17059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:36.959424 17013 server_base.cc:1034] running on GCE node
I20260927 22:55:36.960006 17013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:36.960929 17013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:36.962145 17013 hybrid_clock.cc:648] HybridClock initialized: now 1790549736962118 us; error 43 us; skew 500 ppm
I20260927 22:55:36.962443 17013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:36 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:36.965400 17013 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:55:36.966738 17013 webserver.cc:460] Webserver started at http://127.0.97.1:45757/ using document root <none> and password file <none>
I20260927 22:55:36.967381 17013 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:36.967525 17013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:36.972407 17013 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:55:36.975208 17068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:36.976424 17013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:55:36.976593 17013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "a2541e1068fe4d47907fb2d09336d68e"
format_stamp: "Formatted at 2026-09-27 22:54:51 on dist-test-slave-df53"
server_key: "6b8b726136ea3dd2b880a154f0413e06"
server_key_iv: "5605bc53ec35f6c6a8141883eae117b3"
server_key_version: "encryptionkey@0"
I20260927 22:55:36.977519 17013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:36.996572 17013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:37.000150 17013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:37.000412 17013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:37.001297 17013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:37.002966 17075 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 22:55:37.007881 17075 ts_tablet_manager.cc:1852] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 22:55:37.008849 17075 ts_tablet_manager.cc:1905] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:37.015669 17075 ts_tablet_manager.cc:1918] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:37.015853 17075 log.cc:1196] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/9a159cd9ad4849e7a9bdb56f8b17acb1
I20260927 22:55:37.016350 17075 ts_tablet_manager.cc:1939] T 9a159cd9ad4849e7a9bdb56f8b17acb1 P a2541e1068fe4d47907fb2d09336d68e: Deleting consensus metadata
I20260927 22:55:37.016703 17013 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 22:55:37.016840 17013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260927 22:55:37.016968 17013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:37.017060 17013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:37.050082 17013 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43395
I20260927 22:55:37.050169 17181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43395 every 8 connection(s)
I20260927 22:55:37.051363 17013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:55:37.055958 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:37.054264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:55223)
0927 22:55:37.054960 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:37.055149 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":265,"threads_started":1}
W20260927 22:55:37.056563 17182 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35573 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35573: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
I20260927 22:55:37.057324 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:37.057011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:43077)
0927 22:55:37.057135 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:37.057228 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:55:37.058290 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:37.057933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:44623)
0927 22:55:37.058074 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:37.058156 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:55:37.058514 17182 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:55:37.059065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 17013
I20260927 22:55:37.059631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34455
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35573
--webserver_port=36735
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:55:37.378006 17185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:37.378258 17185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:37.378352 17185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:37.385656 17185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:37.385754 17185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:37.385876 17185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:55:37.395692 17185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36735
--tserver_master_addrs=127.0.97.62:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:37.396874 17185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:37.398157 17185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:37.410782 17190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:37.410707 17191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:37.410741 17193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:37.411882 17185 server_base.cc:1034] running on GCE node
I20260927 22:55:37.412467 17185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:37.413367 17185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:37.414620 17185 hybrid_clock.cc:648] HybridClock initialized: now 1790549737414547 us; error 86 us; skew 500 ppm
I20260927 22:55:37.414956 17185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:37 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:37.417953 17185 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:55:37.419304 17185 webserver.cc:460] Webserver started at http://127.0.97.2:36735/ using document root <none> and password file <none>
I20260927 22:55:37.419972 17185 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:37.420120 17185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:37.425112 17185 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:55:37.427717 17200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:37.428987 17185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:55:37.429178 17185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "a4c1b3b0a0e14a3da8573cc99e9d631a"
format_stamp: "Formatted at 2026-09-27 22:54:52 on dist-test-slave-df53"
server_key: "057d77b1ca92499dcc32c13ce0b181bc"
server_key_iv: "302afed457600533a2f2b4b2eaff57e8"
server_key_version: "encryptionkey@0"
I20260927 22:55:37.430189 17185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:37.446902 17185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:37.450636 17185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:37.450922 17185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:37.451841 17185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:37.453109 17185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:37.453227 17185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:37.453365 17185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:37.453457 17185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:37.487025 17185 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34455
I20260927 22:55:37.487080 17312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34455 every 8 connection(s)
I20260927 22:55:37.488468 17185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:55:37.491894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 17185
I20260927 22:55:37.492487 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40415
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35573
--webserver_port=45091
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:55:37.493479 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:37.491394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:43195)
0927 22:55:37.492332 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:37.492566 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":364,"threads_started":1}
W20260927 22:55:37.494203 17313 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35573 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35573: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
I20260927 22:55:37.495043 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:37.494733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:56341)
0927 22:55:37.494881 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:37.494942 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:55:37.495802 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:37.495484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:59451)
0927 22:55:37.495623 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:37.495694 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:55:37.495997 17313 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:55:37.806944 17316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:37.807189 17316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:37.807283 17316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:37.814342 17316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:37.814436 17316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:37.814543 17316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:55:37.823901 17316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45091
--tserver_master_addrs=127.0.97.62:35573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:37.824991 17316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:37.826241 17316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:37.838881 17322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:37.838845 17321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:37.838918 17324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:37.839567 17316 server_base.cc:1034] running on GCE node
I20260927 22:55:37.840132 17316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:37.840987 17316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:37.842252 17316 hybrid_clock.cc:648] HybridClock initialized: now 1790549737842175 us; error 32 us; skew 500 ppm
I20260927 22:55:37.842551 17316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:37 dist-test-slave-df53 krb5kdc[15862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:37.845499 17316 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:55:37.846810 17316 webserver.cc:460] Webserver started at http://127.0.97.3:45091/ using document root <none> and password file <none>
I20260927 22:55:37.847474 17316 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:37.847618 17316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:37.852528 17316 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 22:55:37.855012 17331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:37.856263 17316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:55:37.856429 17316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "6fb52da8556e4e67aa61167f27e2b281"
format_stamp: "Formatted at 2026-09-27 22:54:52 on dist-test-slave-df53"
server_key: "239bfab5715e27888f10cd28bebab030"
server_key_iv: "ddde63222c2bccfb3aeecb50d4e876f5"
server_key_version: "encryptionkey@0"
I20260927 22:55:37.857342 17316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:37.879540 17316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:37.883641 17316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:37.883905 17316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:37.884857 17316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:37.886193 17316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:37.886312 17316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:37.886451 17316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:37.886548 17316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:37.918819 17316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40415
I20260927 22:55:37.918903 17443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40415 every 8 connection(s)
I20260927 22:55:37.920312 17316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:55:37.924204 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 17316
I20260927 22:55:37.925253 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:37.923204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:38409)
0927 22:55:37.923911 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:37.924121 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":272,"threads_started":1}
W20260927 22:55:37.925907 17444 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35573 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35573: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
I20260927 22:55:37.926679 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:37.926399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:44201)
0927 22:55:37.926521 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:37.926580 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:55:37.927521 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:37.927216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:50799)
0927 22:55:37.927364 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:37.927429 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 22:55:37.927750 17444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:55:38.059731 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:38.058990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:52155)
0927 22:55:38.059432 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:38.059511 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":224,"threads_started":1}
I20260927 22:55:38.497219 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:38.496473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:50021)
0927 22:55:38.496921 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:38.497002 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260927 22:55:38.928931 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:38.928199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:53067)
0927 22:55:38.928643 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:38.928723 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":209,"threads_started":1}
I20260927 22:55:39.061290 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:39.060574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:46759)
0927 22:55:39.061065 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:39.061138 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":305,"threads_started":1}
I20260927 22:55:39.498733 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:39.498012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:41441)
0927 22:55:39.498492 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:39.498580 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":203,"threads_started":1}
I20260927 22:55:39.930563 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:39.929827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:39141)
0927 22:55:39.930290 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:39.930407 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":232,"threads_started":1}
I20260927 22:55:40.062857 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:40.062143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:50569)
0927 22:55:40.062589 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:40.062674 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":220,"threads_started":1}
I20260927 22:55:40.500365 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:40.499698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:57091)
0927 22:55:40.500120 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:40.500202 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260927 22:55:40.932020 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:40.931388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:38437)
0927 22:55:40.931790 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:40.931862 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":221,"threads_started":1}
I20260927 22:55:41.064452 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:41.063781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:38467)
0927 22:55:41.064198 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:41.064275 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":231,"threads_started":1}
I20260927 22:55:41.501929 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:41.501173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:48559)
0927 22:55:41.501672 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:41.501748 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
I20260927 22:55:41.933524 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:41.932771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:38517)
0927 22:55:41.933174 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:41.933262 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":202,"threads_started":1}
I20260927 22:55:42.066166 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:42.065413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:52109)
0927 22:55:42.065927 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:42.066001 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":304,"threads_started":1}
I20260927 22:55:42.503536 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:42.502833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:59883)
0927 22:55:42.503294 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:42.503377 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":251,"threads_started":1}
I20260927 22:55:42.935230 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:42.934483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:51475)
0927 22:55:42.934965 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:42.935057 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":259,"threads_started":1}
I20260927 22:55:43.067646 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:43.066987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:40725)
0927 22:55:43.067423 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:43.067497 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":232,"threads_started":1}
I20260927 22:55:43.505208 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:43.504454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:41931)
0927 22:55:43.504954 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:43.505046 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":271,"threads_started":1}
I20260927 22:55:43.936784 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:43.936101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:54387)
0927 22:55:43.936528 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:43.936632 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":238,"threads_started":1}
I20260927 22:55:44.069198 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:44.068455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:34611)
0927 22:55:44.068933 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:44.069006 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":233,"threads_started":1}
I20260927 22:55:44.506959 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:44.506149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:43827)
0927 22:55:44.506669 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:44.506748 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":212,"threads_started":1}
I20260927 22:55:44.938393 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:44.937573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:33999)
0927 22:55:44.938102 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:44.938176 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":230,"threads_started":1}
I20260927 22:55:45.070788 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:45.070158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:38949)
0927 22:55:45.070556 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:45.070629 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":216,"threads_started":1}
I20260927 22:55:45.508540 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:45.507866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:51021)
0927 22:55:45.508285 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:45.508385 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":208,"threads_started":1}
I20260927 22:55:45.939891 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:45.939221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:52751)
0927 22:55:45.939651 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:45.939727 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":219,"threads_started":1}
I20260927 22:55:46.072391 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:46.071641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:39369)
0927 22:55:46.072129 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:46.072226 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
I20260927 22:55:46.510244 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:46.509460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:51331)
0927 22:55:46.509989 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:46.510071 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":292,"threads_started":1}
I20260927 22:55:46.941468 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:46.940732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:56143)
0927 22:55:46.941195 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:46.941269 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":217,"threads_started":1}
I20260927 22:55:47.074117 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:47.073283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:41415)
0927 22:55:47.073816 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:47.073929 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":251,"threads_started":1}
I20260927 22:55:47.511858 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:47.511175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:60459)
0927 22:55:47.511613 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:47.511689 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":228,"threads_started":1}
I20260927 22:55:47.942956 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:47.942299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:38445)
0927 22:55:47.942717 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:47.942788 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":220,"threads_started":1}
I20260927 22:55:48.075640 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:48.074963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:46419)
0927 22:55:48.075392 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:48.075466 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":244,"threads_started":1}
I20260927 22:55:48.513561 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:48.512856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:48621)
0927 22:55:48.513297 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:48.513385 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":214,"threads_started":1}
I20260927 22:55:48.944473 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:48.943783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:33413)
0927 22:55:48.944193 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:48.944278 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":222,"threads_started":1}
I20260927 22:55:49.077127 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:49.076485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:37501)
0927 22:55:49.076901 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:49.076975 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":217,"threads_started":1}
I20260927 22:55:49.515257 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:49.514503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:59567)
0927 22:55:49.514968 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:49.515047 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":253,"threads_started":1}
I20260927 22:55:49.946122 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:49.945360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:52631)
0927 22:55:49.945882 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:49.945974 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":279,"threads_started":1}
I20260927 22:55:50.078709 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:50.078021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:47367)
0927 22:55:50.078440 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:50.078513 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":216,"threads_started":1}
I20260927 22:55:50.517004 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:50.516271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:38809)
0927 22:55:50.516733 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:50.516816 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":226,"threads_started":1}
I20260927 22:55:50.947711 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:50.947000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:49255)
0927 22:55:50.947449 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:50.947530 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":206,"threads_started":1}
I20260927 22:55:51.080181 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:51.079506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:54043)
0927 22:55:51.079932 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:51.080008 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":211,"threads_started":1}
I20260927 22:55:51.518558 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:51.517867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:53687)
0927 22:55:51.518312 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:51.518389 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":212,"threads_started":1}
I20260927 22:55:51.949268 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:51.948610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:46659)
0927 22:55:51.949025 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:51.949099 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260927 22:55:52.082098 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:52.081300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.1:47767)
0927 22:55:52.081786 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:52.081876 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":228,"threads_started":1}
I20260927 22:55:52.520272 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:52.519589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.2:45897)
0927 22:55:52.519999 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:52.520095 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":209,"threads_started":1}
I20260927 22:55:52.935906 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 17013
I20260927 22:55:52.951103 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:52.950331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35573 (local address 127.0.97.3:40537)
0927 22:55:52.950832 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:52.950930 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":274,"threads_started":1}
I20260927 22:55:52.959407 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:52.936981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.0.97.1:43395)
0927 22:55:52.937683 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:52.937697 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:52.937733 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:55:52.939749 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:52.939759 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:52.940427 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:52.940734 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:52.942217 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:52.943554 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:52.944751 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:52.945406 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:52.945542 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:52.953569 (+ 8027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:52.953577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:52.953652 (+ 75us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:52.953701 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:52.956416 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:52.957055 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:52.957061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:52.957067 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:52.957217 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:52.957558 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:52.957561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:52.957563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:52.958136 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:52.958352 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:52.958699 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 22:55:52.959116 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":302,"threads_started":1}
I20260927 22:55:52.960768 17177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:53.046063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 17013
I20260927 22:55:53.062872 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 17185
I20260927 22:55:53.079543 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:53.063779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.0.97.2:34455)
0927 22:55:53.064357 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:53.064373 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:53.064410 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:55:53.066023 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:53.066033 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:53.066454 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:53.066745 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:53.068048 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:53.069305 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:53.070542 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:53.071200 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:53.071348 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:53.073748 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:53.073755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:53.073769 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:53.073804 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:53.076668 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:53.077243 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:53.077249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:53.077255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:53.077406 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:53.077942 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:53.077945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:53.077947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:53.078436 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:53.078651 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:53.078964 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 22:55:53.079270 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":281,"threads_started":1}
I20260927 22:55:53.080857 17308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:53.163937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 17185
I20260927 22:55:53.181043 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 17316
I20260927 22:55:53.199069 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:53.182052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.0.97.3:40415)
0927 22:55:53.182670 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:53.182684 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:53.182722 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:55:53.184277 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:53.184286 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:53.184715 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:53.184983 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:53.187224 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:53.188422 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:53.189778 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:53.190404 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:53.190595 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:53.193287 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:53.193295 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:53.193308 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:53.193350 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:53.196210 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:53.196815 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:53.196820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:53.196826 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:53.196974 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:53.197289 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:53.197292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:53.197294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:53.197888 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:53.198095 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:53.198391 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 22:55:53.198779 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":331,"threads_started":1}
I20260927 22:55:53.200356 17439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:53.282343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 17316
2026-09-27T22:55:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63918 ms)
[----------] 30 tests from SecurityITest (287944 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:55:53 dist-test-slave-df53 krb5kdc[17501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:55:53 dist-test-slave-df53 krb5kdc[17501](info): set up 2 sockets
Sep 27 22:55:53 dist-test-slave-df53 krb5kdc[17501](info): commencing operation
krb5kdc: starting...
W20260927 22:55:54.401472 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:55:54 dist-test-slave-df53 krb5kdc[17501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:55:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:55:54.626881 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34361
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:55:54.940559 17514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:54.940855 17514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:54.941015 17514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:54.948066 17514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:55:54.948159 17514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:54.948264 17514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:54.948359 17514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:55:54.948446 17514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:55:54.957927 17514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:54.959151 17514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:54.960425 17514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:54.973086 17520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:54.973134 17522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:54.973184 17519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:54.979104 17514 server_base.cc:1034] running on GCE node
I20260927 22:55:54.979843 17514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:54.981199 17514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:54.982427 17514 hybrid_clock.cc:648] HybridClock initialized: now 1790549754982379 us; error 56 us; skew 500 ppm
I20260927 22:55:54.982774 17514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:54 dist-test-slave-df53 krb5kdc[17501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:54.985888 17514 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:55:54.987386 17514 webserver.cc:460] Webserver started at http://127.0.97.62:33283/ using document root <none> and password file <none>
I20260927 22:55:54.988078 17514 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:54.988224 17514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:54.988523 17514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:55:54.991452 17514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "71484d74fbd1428f8a1f73bfa2307f05"
format_stamp: "Formatted at 2026-09-27 22:55:54 on dist-test-slave-df53"
server_key: "91535d345ddb12ef1cadd8ef809c7aa7"
server_key_iv: "8474f32f45ab8d193d05911d596bc2b3"
server_key_version: "encryptionkey@0"
I20260927 22:55:54.992178 17514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "71484d74fbd1428f8a1f73bfa2307f05"
format_stamp: "Formatted at 2026-09-27 22:55:54 on dist-test-slave-df53"
server_key: "91535d345ddb12ef1cadd8ef809c7aa7"
server_key_iv: "8474f32f45ab8d193d05911d596bc2b3"
server_key_version: "encryptionkey@0"
I20260927 22:55:54.997237 17514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:55:55.000497 17529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:55.001776 17514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:55:55.001961 17514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "71484d74fbd1428f8a1f73bfa2307f05"
format_stamp: "Formatted at 2026-09-27 22:55:54 on dist-test-slave-df53"
server_key: "91535d345ddb12ef1cadd8ef809c7aa7"
server_key_iv: "8474f32f45ab8d193d05911d596bc2b3"
server_key_version: "encryptionkey@0"
I20260927 22:55:55.002158 17514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:55.036445 17514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:55.040715 17514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:55.041009 17514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:55.059206 17514 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34361
I20260927 22:55:55.059252 17580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34361 every 8 connection(s)
I20260927 22:55:55.060542 17514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:55:55.064170 17581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:55.065718 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 17514
I20260927 22:55:55.065958 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:55:55.066774 388 external_mini_cluster.cc:1428] Setting key bb79771e77f138c53687f2c5aab6508d
I20260927 22:55:55.078687 17581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05: Bootstrap starting.
Sep 27 22:55:55 dist-test-slave-df53 krb5kdc[17501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:55:55.082327 17581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:55.083485 17581 log.cc:824] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:55.086236 17581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05: No bootstrap required, opened a new log
I20260927 22:55:55.089658 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:55.068760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.0.97.62:34361)
0927 22:55:55.069566 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:55.069581 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:55.069678 (+ 97us) server_negotiation.cc:407] Connection header received
0927 22:55:55.071230 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:55.071262 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:55.071632 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:55.071949 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:55.073719 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:55.075056 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:55.076974 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:55.077668 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:55.077801 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:55.082524 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:55.082533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:55.082547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:55.082580 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:55.085672 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:55.086381 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:55.086388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:55.086393 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:55.086553 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:55.086943 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:55.086950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:55.086959 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:55.087461 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:55.087793 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:55.088250 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 22:55:55.088645 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":405,"threads_started":1}
I20260927 22:55:55.094013 17581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71484d74fbd1428f8a1f73bfa2307f05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34361 } }
I20260927 22:55:55.094374 17581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:55.094475 17581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71484d74fbd1428f8a1f73bfa2307f05, State: Initialized, Role: FOLLOWER
I20260927 22:55:55.094970 17581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71484d74fbd1428f8a1f73bfa2307f05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34361 } }
I20260927 22:55:55.095196 17581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:55:55.095329 17581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:55:55.095515 17581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:55.097836 17581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71484d74fbd1428f8a1f73bfa2307f05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34361 } }
I20260927 22:55:55.098282 17581 leader_election.cc:302] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71484d74fbd1428f8a1f73bfa2307f05; no voters:
I20260927 22:55:55.098830 17581 leader_election.cc:288] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:55:55.098961 17586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:55:55.099658 17586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [term 1 LEADER]: Becoming Leader. State: Replica: 71484d74fbd1428f8a1f73bfa2307f05, State: Running, Role: LEADER
I20260927 22:55:55.100178 17586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71484d74fbd1428f8a1f73bfa2307f05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34361 } }
I20260927 22:55:55.100800 17581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:55:55.104530 17588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71484d74fbd1428f8a1f73bfa2307f05. Latest consensus state: current_term: 1 leader_uuid: "71484d74fbd1428f8a1f73bfa2307f05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71484d74fbd1428f8a1f73bfa2307f05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34361 } } }
I20260927 22:55:55.104902 17588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [sys.catalog]: This master's current role is: LEADER
I20260927 22:55:55.105419 17587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71484d74fbd1428f8a1f73bfa2307f05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71484d74fbd1428f8a1f73bfa2307f05" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34361 } } }
I20260927 22:55:55.105763 17587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05 [sys.catalog]: This master's current role is: LEADER
I20260927 22:55:55.106732 17597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:55:55.110755 17597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:55:55.118709 17597 catalog_manager.cc:1347] Generated new cluster ID: 28f74719d3cf42e488556124efc23889
I20260927 22:55:55.118820 17597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:55:55.136988 17597 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:55:55.138345 17597 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:55:55.144234 17597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05: Generated new TSK 0
I20260927 22:55:55.145231 17597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:55:55.204702 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34361
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:55:55.527298 17607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:55.527595 17607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:55.527751 17607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:55.534931 17607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:55.535027 17607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:55.535240 17607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:55:55.544580 17607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:55.545861 17607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:55.547143 17607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:55.559911 17612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:55.559996 17613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:55.559952 17615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:55.560868 17607 server_base.cc:1034] running on GCE node
I20260927 22:55:55.561476 17607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:55.562369 17607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:55.563560 17607 hybrid_clock.cc:648] HybridClock initialized: now 1790549755563502 us; error 57 us; skew 500 ppm
I20260927 22:55:55.563869 17607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:55 dist-test-slave-df53 krb5kdc[17501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:55.566990 17607 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:55:55.568403 17607 webserver.cc:460] Webserver started at http://127.0.97.1:38921/ using document root <none> and password file <none>
I20260927 22:55:55.569008 17607 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:55.569104 17607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:55.569350 17607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:55:55.572997 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "1992e2be7d9c46119c7fc0d56707a813"
format_stamp: "Formatted at 2026-09-27 22:55:55 on dist-test-slave-df53"
server_key: "19329cb30aca11f63579dc2116b7d8af"
server_key_iv: "ab33965bae14d4f57c285ff5747778c7"
server_key_version: "encryptionkey@0"
I20260927 22:55:55.574103 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1992e2be7d9c46119c7fc0d56707a813"
format_stamp: "Formatted at 2026-09-27 22:55:55 on dist-test-slave-df53"
server_key: "19329cb30aca11f63579dc2116b7d8af"
server_key_iv: "ab33965bae14d4f57c285ff5747778c7"
server_key_version: "encryptionkey@0"
I20260927 22:55:55.580574 17607 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:55:55.583895 17622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:55.585150 17607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:55:55.585314 17607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "1992e2be7d9c46119c7fc0d56707a813"
format_stamp: "Formatted at 2026-09-27 22:55:55 on dist-test-slave-df53"
server_key: "19329cb30aca11f63579dc2116b7d8af"
server_key_iv: "ab33965bae14d4f57c285ff5747778c7"
server_key_version: "encryptionkey@0"
I20260927 22:55:55.585501 17607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:55.608178 17607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:55.612845 17607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:55.613134 17607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:55.614094 17607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:55.615321 17607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:55.615437 17607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:55.615573 17607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:55.615641 17607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:55.647387 17607 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38329
I20260927 22:55:55.647437 17734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38329 every 8 connection(s)
I20260927 22:55:55.648584 17607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:55:55.656312 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 17607
I20260927 22:55:55.656517 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:55:55.657287 388 external_mini_cluster.cc:1428] Setting key 3318b69920e03bdc1f53f60b3c9df285
Sep 27 22:55:55 dist-test-slave-df53 krb5kdc[17501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:55:55.669026 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:55.650923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38803 (local address 127.0.97.62:34361)
0927 22:55:55.651366 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:55.651372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:55.652181 (+ 809us) server_negotiation.cc:407] Connection header received
0927 22:55:55.654078 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:55.654084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:55.654179 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:55.654298 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:55.655801 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:55.656929 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:55.658664 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:55.659183 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:55.659309 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:55.662579 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:55.662588 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:55.662596 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:55.662639 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:55.666069 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:55.666726 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:55.666734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:55.666740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:55.666890 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:55.667284 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:55.667292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:55.667298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:55.667608 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:55.667719 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:55.668618 (+ 899us) server_negotiation.cc:299] Negotiation successful
0927 22:55:55.668778 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":235,"threads_started":1}
I20260927 22:55:55.670410 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:55.651341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34361 (local address 127.0.97.1:38803)
0927 22:55:55.652020 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:55.652056 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:55.653810 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:55.654534 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:55.654542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:55.655039 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:55.655628 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:55.655650 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:55.657089 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:55.657093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:55.658535 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:55.658547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:55.659379 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:55.659383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:55.659566 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:55.660224 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:55:55.660245 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:55.662362 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:55.666198 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:55.666204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:55.666215 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:55.666595 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:55.666999 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:55.667002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:55.667004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:55.667162 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:55.667735 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:55.667744 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:55.668178 (+ 434us) client_negotiation.cc:769] Sending connection context
0927 22:55:55.668515 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 22:55:55.669056 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":259,"threads_started":1}
I20260927 22:55:55.671841 17735 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34361
I20260927 22:55:55.672138 17735 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:55.672792 17735 heartbeater.cc:511] Master 127.0.97.62:34361 requested a full tablet report, sending...
I20260927 22:55:55.674769 17546 ts_manager.cc:194] Registered new tserver with Master: 1992e2be7d9c46119c7fc0d56707a813 (127.0.97.1:38329)
I20260927 22:55:55.676136 17546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38803
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:55:55.710796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34361
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:55:56.020134 17741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:56.020383 17741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:56.020540 17741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:56.027439 17741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:56.027526 17741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:56.027725 17741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:55:56.036851 17741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:56.038051 17741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:56.039292 17741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:56.051749 17747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:56.051720 17746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:56.051834 17749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:56.052820 17741 server_base.cc:1034] running on GCE node
I20260927 22:55:56.053428 17741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:56.054350 17741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:56.055552 17741 hybrid_clock.cc:648] HybridClock initialized: now 1790549756055514 us; error 49 us; skew 500 ppm
I20260927 22:55:56.055861 17741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:56 dist-test-slave-df53 krb5kdc[17501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:56.059063 17741 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:55:56.060547 17741 webserver.cc:460] Webserver started at http://127.0.97.2:44949/ using document root <none> and password file <none>
I20260927 22:55:56.061206 17741 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:56.061328 17741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:56.061702 17741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:55:56.064450 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "70f05bc9e21b479c95381d2d10b45d80"
format_stamp: "Formatted at 2026-09-27 22:55:56 on dist-test-slave-df53"
server_key: "6a41ade4b405698c73aec8cb102b6cbf"
server_key_iv: "6d4a3172a89603b69fd38007fed37573"
server_key_version: "encryptionkey@0"
I20260927 22:55:56.065205 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "70f05bc9e21b479c95381d2d10b45d80"
format_stamp: "Formatted at 2026-09-27 22:55:56 on dist-test-slave-df53"
server_key: "6a41ade4b405698c73aec8cb102b6cbf"
server_key_iv: "6d4a3172a89603b69fd38007fed37573"
server_key_version: "encryptionkey@0"
I20260927 22:55:56.070250 17741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 22:55:56.073560 17756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:56.074836 17741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:55:56.074996 17741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "70f05bc9e21b479c95381d2d10b45d80"
format_stamp: "Formatted at 2026-09-27 22:55:56 on dist-test-slave-df53"
server_key: "6a41ade4b405698c73aec8cb102b6cbf"
server_key_iv: "6d4a3172a89603b69fd38007fed37573"
server_key_version: "encryptionkey@0"
I20260927 22:55:56.075187 17741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:56.106652 17741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:56.110785 17741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:56.111070 17741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:56.112033 17741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:56.113291 17741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:56.113395 17741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:56.113585 17741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:56.113672 17741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 22:55:56.145773 17741 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39149
I20260927 22:55:56.145828 17868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39149 every 8 connection(s)
I20260927 22:55:56.147032 17741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:55:56.150908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 17741
I20260927 22:55:56.151088 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:55:56.151651 388 external_mini_cluster.cc:1428] Setting key 406b87ce9e2f43a65984e2e13a014695
Sep 27 22:55:56 dist-test-slave-df53 krb5kdc[17501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:55:56.167313 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.149530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41299 (local address 127.0.97.62:34361)
0927 22:55:56.149745 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:56.149749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:56.151012 (+ 1263us) server_negotiation.cc:407] Connection header received
0927 22:55:56.153251 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:56.153255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:56.153315 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:56.153423 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:56.154976 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.155715 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.157233 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.157717 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.157848 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.161085 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:56.161095 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:56.161100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:56.161143 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:56.164000 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:56.165009 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:56.165013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:56.165016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:56.165135 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:56.165665 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:56.165668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:56.165670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:56.165899 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:56.166007 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:56.166909 (+ 902us) server_negotiation.cc:299] Negotiation successful
0927 22:55:56.167069 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 22:55:56.168980 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.150031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34361 (local address 127.0.97.2:41299)
0927 22:55:56.150807 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:56.150875 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:56.152979 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:56.153709 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:56.153721 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:56.154267 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:56.154803 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.154819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.155879 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.155883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.157079 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.157090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.157941 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.157947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.158158 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.158807 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:55:56.158832 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:56.160868 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:56.164159 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:56.164172 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:56.164192 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:56.164890 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:56.165249 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:56.165257 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:56.165265 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:56.165518 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:56.166030 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:56.166041 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:56.166520 (+ 479us) client_negotiation.cc:769] Sending connection context
0927 22:55:56.166846 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 22:55:56.167423 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":319,"threads_started":1}
I20260927 22:55:56.170490 17869 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34361
I20260927 22:55:56.170818 17869 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:56.171728 17869 heartbeater.cc:511] Master 127.0.97.62:34361 requested a full tablet report, sending...
I20260927 22:55:56.173434 17546 ts_manager.cc:194] Registered new tserver with Master: 70f05bc9e21b479c95381d2d10b45d80 (127.0.97.2:39149)
I20260927 22:55:56.174836 17546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41299
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:55:56.209401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34361
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:55:56.521483 17874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:56.521749 17874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:56.521924 17874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:56.529039 17874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:56.529173 17874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:56.529372 17874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:55:56.538762 17874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:56.539939 17874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:56.541172 17874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:56.554026 17882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:56.554069 17879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:56.554097 17880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:56.554909 17874 server_base.cc:1034] running on GCE node
I20260927 22:55:56.555423 17874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:56.556545 17874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:56.557755 17874 hybrid_clock.cc:648] HybridClock initialized: now 1790549756557711 us; error 33 us; skew 500 ppm
I20260927 22:55:56.558074 17874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:56 dist-test-slave-df53 krb5kdc[17501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:56.561043 17874 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:55:56.562299 17874 webserver.cc:460] Webserver started at http://127.0.97.3:44209/ using document root <none> and password file <none>
I20260927 22:55:56.562906 17874 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:56.562994 17874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:56.563223 17874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:55:56.565914 17874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "180b58d3960c4cc3b7737b877b8b6c8c"
format_stamp: "Formatted at 2026-09-27 22:55:56 on dist-test-slave-df53"
server_key: "f90601e125cd7bc86acee87ccea24fb6"
server_key_iv: "0384b6b72fe2859f77bec0cb30964961"
server_key_version: "encryptionkey@0"
I20260927 22:55:56.566608 17874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "180b58d3960c4cc3b7737b877b8b6c8c"
format_stamp: "Formatted at 2026-09-27 22:55:56 on dist-test-slave-df53"
server_key: "f90601e125cd7bc86acee87ccea24fb6"
server_key_iv: "0384b6b72fe2859f77bec0cb30964961"
server_key_version: "encryptionkey@0"
I20260927 22:55:56.571342 17874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:55:56.574626 17889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:56.576100 17874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:55:56.576227 17874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "180b58d3960c4cc3b7737b877b8b6c8c"
format_stamp: "Formatted at 2026-09-27 22:55:56 on dist-test-slave-df53"
server_key: "f90601e125cd7bc86acee87ccea24fb6"
server_key_iv: "0384b6b72fe2859f77bec0cb30964961"
server_key_version: "encryptionkey@0"
I20260927 22:55:56.576392 17874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:56.588754 17874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:56.593036 17874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:56.593318 17874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:56.594252 17874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:56.595491 17874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:56.595559 17874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:56.595723 17874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:56.595775 17874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:56.627210 17874 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33875
I20260927 22:55:56.627249 18001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33875 every 8 connection(s)
I20260927 22:55:56.628428 17874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:55:56.629230 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 17874
I20260927 22:55:56.629492 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:55:56.630091 388 external_mini_cluster.cc:1428] Setting key d32c2bcb0fe751e240e4c256e488659c
Sep 27 22:55:56 dist-test-slave-df53 krb5kdc[17501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:55:56.648315 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.631139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49861 (local address 127.0.97.62:34361)
0927 22:55:56.631377 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:56.631383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:56.632815 (+ 1432us) server_negotiation.cc:407] Connection header received
0927 22:55:56.634743 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:56.634747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:56.634808 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:56.634895 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:56.636190 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.636975 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.638550 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.638914 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.639004 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.642120 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:56.642124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:56.642127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:56.642155 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:56.645021 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:56.645938 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:56.645941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:56.645943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:56.646066 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:56.646487 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:56.646492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:56.646494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:56.646852 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:56.647016 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:56.647732 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 22:55:56.647963 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:55:56.648924 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.631796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34361 (local address 127.0.97.3:49861)
0927 22:55:56.632654 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:56.632703 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:56.634497 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:56.635073 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:56.635081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:56.635547 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:56.636054 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.636069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.637119 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.637123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.638407 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.638419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.639082 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.639086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.639289 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.639871 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:55:56.639891 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:56.641943 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:56.645187 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:56.645194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:56.645206 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:56.645822 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:56.646180 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:56.646184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:56.646186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:56.646358 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:56.647051 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:56.647057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:56.647342 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 22:55:56.647560 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 22:55:56.647899 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":362,"threads_started":1}
I20260927 22:55:56.650238 18002 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34361
I20260927 22:55:56.650529 18002 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:56.651185 18002 heartbeater.cc:511] Master 127.0.97.62:34361 requested a full tablet report, sending...
I20260927 22:55:56.652514 17546 ts_manager.cc:194] Registered new tserver with Master: 180b58d3960c4cc3b7737b877b8b6c8c (127.0.97.3:33875)
I20260927 22:55:56.653271 17546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49861
I20260927 22:55:56.657549 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:55:56.674150 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.661928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.0.97.62:34361)
0927 22:55:56.662123 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:56.662130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:56.662476 (+ 346us) server_negotiation.cc:407] Connection header received
0927 22:55:56.662673 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:56.662677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:56.662743 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:56.662842 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:56.664270 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.664984 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.666341 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.666749 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.666891 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.669138 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:56.669144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:56.669149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:56.669186 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:56.671574 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:56.672241 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:56.672246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:56.672250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:56.672372 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:56.672750 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:56.672757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:56.672760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:56.673045 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:56.673287 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:56.673635 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 22:55:56.673852 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:55:56.679003 17735 heartbeater.cc:503] Master 127.0.97.62:34361 was elected leader, sending a full tablet report...
I20260927 22:55:56.677867 17546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:55:56.685267 17546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:55:56.719732 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.696944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.0.97.2:39149)
0927 22:55:56.710862 (+ 13918us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:56.710872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:56.710912 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:55:56.711004 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:56.711013 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:56.711337 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:56.711561 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:56.712132 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.713777 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.717204 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.718241 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.718315 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.718516 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:56.719241 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 22:55:56.719442 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13742,"thread_start_us":13645,"threads_started":1}
I20260927 22:55:56.719827 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.696773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39149 (local address 127.0.0.1:60938)
0927 22:55:56.709083 (+ 12310us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:56.709132 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:56.709336 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:56.711583 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:56.711590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:56.711618 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:56.711954 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.711972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.713963 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.713971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.717026 (+ 3055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.717038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.718592 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.718598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.718929 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.719058 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 22:55:56.719196 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 22:55:56.719388 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11973,"mutex_wait_us":63,"thread_start_us":14189,"threads_started":1}
I20260927 22:55:56.721405 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.697062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38329 (local address 127.0.0.1:49196)
0927 22:55:56.711249 (+ 14187us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:56.711281 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:56.711405 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:56.713649 (+ 2244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:56.713656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:56.713692 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:56.714306 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.714321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.716626 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.716630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.718799 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.718813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.720584 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.720591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.720746 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.720776 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:55:56.720862 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 22:55:56.721010 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14039,"mutex_wait_us":1,"thread_start_us":12488,"threads_started":1}
I20260927 22:55:56.723335 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.712639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33875 (local address 127.0.0.1:48710)
0927 22:55:56.715313 (+ 2674us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:56.715339 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:56.715460 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:56.716019 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:56.716024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:56.716048 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:56.716384 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.716394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.718442 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.718447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.720986 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.720997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.722835 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.722839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.722949 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.722969 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 22:55:56.723040 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:55:56.723112 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2530,"mutex_wait_us":20,"thread_start_us":2360,"threads_started":1}
I20260927 22:55:56.723563 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.711673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.0.97.3:33875)
0927 22:55:56.714743 (+ 3070us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:56.714755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:56.715449 (+ 694us) server_negotiation.cc:407] Connection header received
0927 22:55:56.715621 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:56.715628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:56.715821 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:56.716016 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:56.716616 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.718285 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.721161 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.722639 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.722770 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.722966 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:56.723099 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 22:55:56.723240 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2866,"thread_start_us":2698,"threads_started":1}
I20260927 22:55:56.724078 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.697283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.0.97.1:38329)
0927 22:55:56.712702 (+ 15419us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:56.712713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:56.712744 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:55:56.712859 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:56.712866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:56.713114 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:56.713335 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:56.715069 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.716477 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.718968 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.720368 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.720487 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.720702 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:56.723718 (+ 3016us) server_negotiation.cc:299] Negotiation successful
0927 22:55:56.723858 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15212,"thread_start_us":14997,"threads_started":1}
I20260927 22:55:56.724109 17804 tablet_service.cc:1467] Processing CreateTablet for tablet 412d52d34ffc4ee9abf0c27b5631395b (DEFAULT_TABLE table=test-table [id=c1c3e58326d14ecb83cb0208dbffbc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:56.725493 17804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412d52d34ffc4ee9abf0c27b5631395b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:56.726575 17937 tablet_service.cc:1467] Processing CreateTablet for tablet 412d52d34ffc4ee9abf0c27b5631395b (DEFAULT_TABLE table=test-table [id=c1c3e58326d14ecb83cb0208dbffbc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:56.727746 17937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412d52d34ffc4ee9abf0c27b5631395b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:56.727502 17670 tablet_service.cc:1467] Processing CreateTablet for tablet 412d52d34ffc4ee9abf0c27b5631395b (DEFAULT_TABLE table=test-table [id=c1c3e58326d14ecb83cb0208dbffbc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:56.728763 17670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412d52d34ffc4ee9abf0c27b5631395b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:56.737646 18020 tablet_bootstrap.cc:492] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: Bootstrap starting.
I20260927 22:55:56.740671 18020 tablet_bootstrap.cc:654] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:56.741714 18020 log.cc:824] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:56.746142 18023 tablet_bootstrap.cc:492] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: Bootstrap starting.
I20260927 22:55:56.746387 18021 tablet_bootstrap.cc:492] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: Bootstrap starting.
I20260927 22:55:56.750412 18021 tablet_bootstrap.cc:654] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:56.750587 18023 tablet_bootstrap.cc:654] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:56.751570 18023 log.cc:824] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:56.751659 18021 log.cc:824] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:56.752449 18020 tablet_bootstrap.cc:492] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: No bootstrap required, opened a new log
I20260927 22:55:56.752817 18020 ts_tablet_manager.cc:1397] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20260927 22:55:56.754396 18023 tablet_bootstrap.cc:492] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: No bootstrap required, opened a new log
I20260927 22:55:56.754633 18023 ts_tablet_manager.cc:1397] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 22:55:56.756099 18021 tablet_bootstrap.cc:492] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: No bootstrap required, opened a new log
I20260927 22:55:56.756384 18021 ts_tablet_manager.cc:1397] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20260927 22:55:56.762223 18023 raft_consensus.cc:348] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } }
I20260927 22:55:56.762633 18023 raft_consensus.cc:374] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:56.762740 18023 raft_consensus.cc:729] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1992e2be7d9c46119c7fc0d56707a813, State: Initialized, Role: FOLLOWER
I20260927 22:55:56.763350 18023 consensus_queue.cc:260] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } }
I20260927 22:55:56.763535 18020 raft_consensus.cc:348] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } }
I20260927 22:55:56.764003 18020 raft_consensus.cc:374] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:56.764128 18020 raft_consensus.cc:729] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180b58d3960c4cc3b7737b877b8b6c8c, State: Initialized, Role: FOLLOWER
I20260927 22:55:56.764731 18020 consensus_queue.cc:260] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } }
I20260927 22:55:56.765811 18002 heartbeater.cc:503] Master 127.0.97.62:34361 was elected leader, sending a full tablet report...
I20260927 22:55:56.766481 18020 ts_tablet_manager.cc:1428] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260927 22:55:56.766636 18023 ts_tablet_manager.cc:1428] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260927 22:55:56.767995 18021 raft_consensus.cc:348] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } }
I20260927 22:55:56.768431 18021 raft_consensus.cc:374] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:56.768537 18021 raft_consensus.cc:729] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f05bc9e21b479c95381d2d10b45d80, State: Initialized, Role: FOLLOWER
I20260927 22:55:56.769126 18021 consensus_queue.cc:260] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } }
I20260927 22:55:56.770524 17869 heartbeater.cc:503] Master 127.0.97.62:34361 was elected leader, sending a full tablet report...
I20260927 22:55:56.771159 18021 ts_tablet_manager.cc:1428] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: Time spent starting tablet: real 0.015s user 0.011s sys 0.001s
W20260927 22:55:56.880146 18003 tablet.cc:2369] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:55:56.899062 17870 tablet.cc:2369] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:55:56.901028 17736 tablet.cc:2369] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:55:56.989209 18026 raft_consensus.cc:482] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:55:56.989529 18026 raft_consensus.cc:504] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } }
I20260927 22:55:56.991393 18026 leader_election.cc:288] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70f05bc9e21b479c95381d2d10b45d80 (127.0.97.2:39149), 180b58d3960c4cc3b7737b877b8b6c8c (127.0.97.3:33875)
I20260927 22:55:56.995664 18027 raft_consensus.cc:482] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:55:56.995929 18027 raft_consensus.cc:504] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } }
I20260927 22:55:56.998576 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.991903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43315 (local address 127.0.97.2:39149)
0927 22:55:56.992133 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:56.992139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:56.992339 (+ 200us) server_negotiation.cc:407] Connection header received
0927 22:55:56.992563 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:56.992572 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:56.992663 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:56.992760 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:56.993479 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.994422 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.996612 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.997532 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.997692 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.997793 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:56.998163 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 22:55:56.998311 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:55:56.998600 18027 leader_election.cc:288] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1992e2be7d9c46119c7fc0d56707a813 (127.0.97.1:38329), 70f05bc9e21b479c95381d2d10b45d80 (127.0.97.2:39149)
I20260927 22:55:56.999291 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.991633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39149 (local address 127.0.97.1:43315)
0927 22:55:56.992205 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:56.992221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:56.992399 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:56.992867 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:56.992872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:56.992904 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:56.993304 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.993317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.994542 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.994547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.996467 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.996479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.997751 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.997759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.997960 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.997994 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:55:56.998737 (+ 743us) client_negotiation.cc:240] Negotiation successful
0927 22:55:56.998910 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":284,"threads_started":1}
I20260927 22:55:56.999887 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.992670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39361 (local address 127.0.97.3:33875)
0927 22:55:56.992863 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:56.992869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:56.993069 (+ 200us) server_negotiation.cc:407] Connection header received
0927 22:55:56.993259 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:56.993266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:56.993341 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:56.993433 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:56.994026 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.995021 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.997340 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.998616 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.998734 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.998830 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:56.999356 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 22:55:56.999658 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:55:57.000586 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.992366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33875 (local address 127.0.97.1:39361)
0927 22:55:56.992979 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:56.992992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:56.993134 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:56.993473 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:56.993477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:56.993501 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:56.993886 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.993898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.995160 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.995163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.997174 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.997187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:56.998977 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:56.998982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:56.999159 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:56.999205 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:55:56.999912 (+ 707us) client_negotiation.cc:240] Negotiation successful
0927 22:55:57.000047 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"spinlock_wait_cycles":16640,"thread_start_us":274,"threads_started":1}
I20260927 22:55:57.000501 17824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412d52d34ffc4ee9abf0c27b5631395b" candidate_uuid: "1992e2be7d9c46119c7fc0d56707a813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f05bc9e21b479c95381d2d10b45d80" is_pre_election: true
I20260927 22:55:57.001078 17824 raft_consensus.cc:2454] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1992e2be7d9c46119c7fc0d56707a813 in term 0.
I20260927 22:55:57.001905 17626 leader_election.cc:302] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1992e2be7d9c46119c7fc0d56707a813, 70f05bc9e21b479c95381d2d10b45d80; no voters:
I20260927 22:55:57.001916 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412d52d34ffc4ee9abf0c27b5631395b" candidate_uuid: "1992e2be7d9c46119c7fc0d56707a813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" is_pre_election: true
I20260927 22:55:57.002427 17957 raft_consensus.cc:2454] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1992e2be7d9c46119c7fc0d56707a813 in term 0.
I20260927 22:55:57.004130 18026 raft_consensus.cc:2781] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:55:57.004292 18026 raft_consensus.cc:482] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:55:57.004437 18026 raft_consensus.cc:3037] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:57.005461 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.997925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38329 (local address 127.0.97.3:34095)
0927 22:55:56.998120 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:56.998138 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:56.998234 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:56.998857 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:56.998863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:56.998909 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:56.999134 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:56.999144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.000739 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.000743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:57.002873 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:57.002900 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.004668 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.004675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:57.004911 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:57.004946 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:55:57.005042 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:55:57.005186 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 22:55:57.005452 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.998320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34095 (local address 127.0.97.1:38329)
0927 22:55:56.998501 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:56.998507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:56.998523 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:55:56.998574 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:56.998579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:56.998646 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:56.998724 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:56.999296 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.000590 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.003231 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.004497 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.004628 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:57.004723 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:57.005091 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 22:55:57.005230 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:55:57.006739 17690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412d52d34ffc4ee9abf0c27b5631395b" candidate_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1992e2be7d9c46119c7fc0d56707a813" is_pre_election: true
I20260927 22:55:57.007511 18026 raft_consensus.cc:504] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } }
I20260927 22:55:57.007963 17690 raft_consensus.cc:2379] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 180b58d3960c4cc3b7737b877b8b6c8c in current term 1: Already voted for candidate 1992e2be7d9c46119c7fc0d56707a813 in this term.
I20260927 22:55:57.008332 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.998981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39149 (local address 127.0.97.3:42169)
0927 22:55:57.000601 (+ 1620us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:57.000619 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:57.000767 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:57.001881 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:57.001885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:57.001914 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:57.002259 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:57.002272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.003897 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.003902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:57.005889 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:57.005923 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.007560 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.007565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:57.007736 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:57.007773 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:55:57.007871 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:55:57.008002 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1516,"thread_start_us":2004,"threads_started":1}
I20260927 22:55:57.008484 18026 leader_election.cc:288] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [CANDIDATE]: Term 1 election: Requested vote from peers 70f05bc9e21b479c95381d2d10b45d80 (127.0.97.2:39149), 180b58d3960c4cc3b7737b877b8b6c8c (127.0.97.3:33875)
I20260927 22:55:57.008474 17824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412d52d34ffc4ee9abf0c27b5631395b" candidate_uuid: "1992e2be7d9c46119c7fc0d56707a813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f05bc9e21b479c95381d2d10b45d80"
I20260927 22:55:57.008684 17824 raft_consensus.cc:3037] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:57.008894 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:56.999401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42169 (local address 127.0.97.2:39149)
0927 22:55:56.999581 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:56.999588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:57.000681 (+ 1093us) server_negotiation.cc:407] Connection header received
0927 22:55:57.001313 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:57.001319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:57.001403 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:57.001705 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:57.002520 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.003726 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.006079 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.007224 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.007318 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:57.007409 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:57.007914 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 22:55:57.008046 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:55:57.008867 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412d52d34ffc4ee9abf0c27b5631395b" candidate_uuid: "1992e2be7d9c46119c7fc0d56707a813" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "180b58d3960c4cc3b7737b877b8b6c8c"
I20260927 22:55:57.009054 17957 raft_consensus.cc:3037] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:57.009610 17823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412d52d34ffc4ee9abf0c27b5631395b" candidate_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f05bc9e21b479c95381d2d10b45d80" is_pre_election: true
I20260927 22:55:57.010488 17893 leader_election.cc:302] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 180b58d3960c4cc3b7737b877b8b6c8c; no voters: 1992e2be7d9c46119c7fc0d56707a813, 70f05bc9e21b479c95381d2d10b45d80
I20260927 22:55:57.011461 17824 raft_consensus.cc:2454] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1992e2be7d9c46119c7fc0d56707a813 in term 1.
I20260927 22:55:57.011843 17626 leader_election.cc:302] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1992e2be7d9c46119c7fc0d56707a813, 70f05bc9e21b479c95381d2d10b45d80; no voters:
I20260927 22:55:57.012023 17957 raft_consensus.cc:2454] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1992e2be7d9c46119c7fc0d56707a813 in term 1.
I20260927 22:55:57.012146 18026 raft_consensus.cc:2781] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:55:57.012214 18027 raft_consensus.cc:2726] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:55:57.012610 18026 raft_consensus.cc:686] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 1 LEADER]: Becoming Leader. State: Replica: 1992e2be7d9c46119c7fc0d56707a813, State: Running, Role: LEADER
I20260927 22:55:57.013094 18026 consensus_queue.cc:237] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } }
I20260927 22:55:57.017239 17545 catalog_manager.cc:5495] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1992e2be7d9c46119c7fc0d56707a813 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1992e2be7d9c46119c7fc0d56707a813" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1992e2be7d9c46119c7fc0d56707a813" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 } health_report { overall_health: UNKNOWN } } }
I20260927 22:55:57.086416 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:57.081096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.0.97.1:38329)
0927 22:55:57.081290 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:57.081296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:57.081316 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:55:57.081517 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:57.081521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:57.081631 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:57.081785 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:55:57.082157 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.082906 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.084316 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.084727 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.084860 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:57.085337 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:55:57.085837 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:55:57.086012 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:57.086061 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 22:55:57.086137 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:55:57.094316 17824 raft_consensus.cc:1261] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80 [term 1 FOLLOWER]: Refusing update from remote peer 1992e2be7d9c46119c7fc0d56707a813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:55:57.094345 17957 raft_consensus.cc:1261] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 1 FOLLOWER]: Refusing update from remote peer 1992e2be7d9c46119c7fc0d56707a813: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:55:57.095166 18032 consensus_queue.cc:1035] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70f05bc9e21b479c95381d2d10b45d80" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:55:57.095542 18026 consensus_queue.cc:1035] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [LEADER]: Connected to new peer: Peer: permanent_uuid: "180b58d3960c4cc3b7737b877b8b6c8c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:55:57.116968 18037 mvcc.cc:205] Tried to move back new op lower bound from 7334091805048778752 to 7334091804729163776. Current Snapshot: MvccSnapshot[applied={T|T < 7334091805048778752}]
I20260927 22:55:57.122879 17545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:55:57.123190 17545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:55:57.127022 18036 mvcc.cc:205] Tried to move back new op lower bound from 7334091805048778752 to 7334091804729163776. Current Snapshot: MvccSnapshot[applied={T|T < 7334091805048778752}]
I20260927 22:55:57.127851 17545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71484d74fbd1428f8a1f73bfa2307f05: Sending DeleteTablet for 3 replicas of tablet 412d52d34ffc4ee9abf0c27b5631395b
I20260927 22:55:57.130434 17937 tablet_service.cc:1514] Processing DeleteTablet for tablet 412d52d34ffc4ee9abf0c27b5631395b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48710
I20260927 22:55:57.130736 17804 tablet_service.cc:1514] Processing DeleteTablet for tablet 412d52d34ffc4ee9abf0c27b5631395b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60938
I20260927 22:55:57.132306 17670 tablet_service.cc:1514] Processing DeleteTablet for tablet 412d52d34ffc4ee9abf0c27b5631395b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49196
I20260927 22:55:57.133448 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 17607
I20260927 22:55:57.142396 18046 tablet_replica.cc:330] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: stopping tablet replica
I20260927 22:55:57.142874 18046 raft_consensus.cc:2229] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:55:57.143445 18046 raft_consensus.cc:2258] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:57.145648 18046 ts_tablet_manager.cc:1905] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:57.152422 18046 ts_tablet_manager.cc:1918] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:57.152593 18046 log.cc:1196] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/412d52d34ffc4ee9abf0c27b5631395b
I20260927 22:55:57.153108 18046 ts_tablet_manager.cc:1939] T 412d52d34ffc4ee9abf0c27b5631395b P 180b58d3960c4cc3b7737b877b8b6c8c: Deleting consensus metadata
I20260927 22:55:57.154804 17533 catalog_manager.cc:4841] TS 180b58d3960c4cc3b7737b877b8b6c8c (127.0.97.3:33875): tablet 412d52d34ffc4ee9abf0c27b5631395b (table test-table [id=c1c3e58326d14ecb83cb0208dbffbc1f]) successfully deleted
I20260927 22:55:57.161346 18048 tablet_replica.cc:330] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: stopping tablet replica
I20260927 22:55:57.164080 18048 raft_consensus.cc:2229] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:55:57.164753 18047 tablet_replica.cc:330] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: stopping tablet replica
I20260927 22:55:57.165467 18048 raft_consensus.cc:2258] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:57.165979 18047 raft_consensus.cc:2229] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:55:57.168361 18048 ts_tablet_manager.cc:1905] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:55:57 dist-test-slave-df53 krb5kdc[17501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:55:57.172906 18047 raft_consensus.cc:2258] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:57.175022 18047 ts_tablet_manager.cc:1905] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:57.179421 18048 ts_tablet_manager.cc:1918] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:57.179705 18048 log.cc:1196] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/412d52d34ffc4ee9abf0c27b5631395b
I20260927 22:55:57.180348 18048 ts_tablet_manager.cc:1939] T 412d52d34ffc4ee9abf0c27b5631395b P 1992e2be7d9c46119c7fc0d56707a813: Deleting consensus metadata
I20260927 22:55:57.181778 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:57.134770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.0.97.1:38329)
0927 22:55:57.134964 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:57.134972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:57.160637 (+ 25665us) server_negotiation.cc:407] Connection header received
0927 22:55:57.161796 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:57.161802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:57.161882 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:57.161955 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:57.164666 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.165865 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.167759 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.168410 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.168516 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:57.173771 (+ 5255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:57.173787 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:57.173804 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:57.173850 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:57.177919 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:57.178808 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:57.178816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:57.178821 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:57.178965 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:57.179533 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:57.179540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:57.179545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:57.180141 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:57.180502 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:57.181192 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 22:55:57.181430 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:55:57.182083 18047 ts_tablet_manager.cc:1918] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:57.182615 18047 log.cc:1196] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/412d52d34ffc4ee9abf0c27b5631395b
I20260927 22:55:57.182544 17532 catalog_manager.cc:4841] TS 1992e2be7d9c46119c7fc0d56707a813 (127.0.97.1:38329): tablet 412d52d34ffc4ee9abf0c27b5631395b (table test-table [id=c1c3e58326d14ecb83cb0208dbffbc1f]) successfully deleted
I20260927 22:55:57.183113 17730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:57.183213 18047 ts_tablet_manager.cc:1939] T 412d52d34ffc4ee9abf0c27b5631395b P 70f05bc9e21b479c95381d2d10b45d80: Deleting consensus metadata
I20260927 22:55:57.184984 17533 catalog_manager.cc:4841] TS 70f05bc9e21b479c95381d2d10b45d80 (127.0.97.2:39149): tablet 412d52d34ffc4ee9abf0c27b5631395b (table test-table [id=c1c3e58326d14ecb83cb0208dbffbc1f]) successfully deleted
I20260927 22:55:57.281023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 17607
I20260927 22:55:57.299286 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 17741
Sep 27 22:55:57 dist-test-slave-df53 krb5kdc[17501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:55:57.314041 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:57.300015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.0.97.2:39149)
0927 22:55:57.300197 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:57.300202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:57.300220 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:55:57.300383 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:57.300387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:57.300457 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:57.300540 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:57.302042 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.302780 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.304212 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.304579 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.304665 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:57.308866 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:57.308877 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:57.308892 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:57.308924 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:57.311691 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:57.312245 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:57.312251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:57.312254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:57.312407 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:57.312705 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:57.312708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:57.312710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:57.313018 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:57.313245 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:57.313457 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 22:55:57.313770 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:55:57.314648 17864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:57.404918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 17741
I20260927 22:55:57.423532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 17874
Sep 27 22:55:57 dist-test-slave-df53 krb5kdc[17501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:55:57.438694 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:57.424335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.0.97.3:33875)
0927 22:55:57.424538 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:57.424543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:57.424560 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:55:57.424706 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:57.424710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:57.424774 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:57.424859 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:57.426221 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.426954 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.428190 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:57.428575 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:57.428667 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:57.432929 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:57.432943 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:57.432958 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:57.432988 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:57.435758 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:57.436619 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:57.436626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:57.436629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:57.436771 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:57.437116 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:57.437120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:57.437122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:57.437477 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:57.437760 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:57.438154 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 22:55:57.438308 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:55:57.439338 17997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:57.527901 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 17874
I20260927 22:55:57.544121 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 17514
I20260927 22:55:57.545197 17576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:57.657505 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 17514
2026-09-27T22:55:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4378 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T22:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:55:57Z Disabled control of system clock
I20260927 22:55:57.730693 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40455
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:55:58.045704 18057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:58.045967 18057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:58.046089 18057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:58.053015 18057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:55:58.053125 18057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:58.053215 18057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:58.053334 18057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:55:58.053411 18057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:55:58.062295 18057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:58.063530 18057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:58.065313 18057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:58.077009 18065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:58.076930 18063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:58.077319 18062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:58.078105 18057 server_base.cc:1034] running on GCE node
I20260927 22:55:58.078836 18057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:58.080211 18057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:58.081425 18057 hybrid_clock.cc:648] HybridClock initialized: now 1790549758081379 us; error 38 us; skew 500 ppm
I20260927 22:55:58.081810 18057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:55:58.087010 18057 webserver.cc:460] Webserver started at http://127.0.97.62:38811/ using document root <none> and password file <none>
I20260927 22:55:58.087683 18057 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:58.087769 18057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:58.088089 18057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:55:58.090967 18057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "e48cef7754024b03ae1bc2a81242b8be"
format_stamp: "Formatted at 2026-09-27 22:55:58 on dist-test-slave-df53"
server_key: "696ab58c22334a5cbb750b1ecd6c4a40"
server_key_iv: "e9518cac942570d7fe3722dca2920d37"
server_key_version: "encryptionkey@0"
I20260927 22:55:58.091708 18057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "e48cef7754024b03ae1bc2a81242b8be"
format_stamp: "Formatted at 2026-09-27 22:55:58 on dist-test-slave-df53"
server_key: "696ab58c22334a5cbb750b1ecd6c4a40"
server_key_iv: "e9518cac942570d7fe3722dca2920d37"
server_key_version: "encryptionkey@0"
I20260927 22:55:58.096565 18057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:55:58.099943 18071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:58.101401 18057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:55:58.101588 18057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "e48cef7754024b03ae1bc2a81242b8be"
format_stamp: "Formatted at 2026-09-27 22:55:58 on dist-test-slave-df53"
server_key: "696ab58c22334a5cbb750b1ecd6c4a40"
server_key_iv: "e9518cac942570d7fe3722dca2920d37"
server_key_version: "encryptionkey@0"
I20260927 22:55:58.101835 18057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:58.135892 18057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:58.136693 18057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:58.137030 18057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:58.155234 18057 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40455
I20260927 22:55:58.155283 18122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40455 every 8 connection(s)
I20260927 22:55:58.156494 18057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:55:58.159588 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 18057
I20260927 22:55:58.159767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:55:58.160439 18123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:58.160440 388 external_mini_cluster.cc:1428] Setting key 43409fa608196076915f2134e746606a
I20260927 22:55:58.173065 18123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be: Bootstrap starting.
I20260927 22:55:58.173548 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:58.162386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.0.97.62:40455)
0927 22:55:58.163398 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:58.163409 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:58.163450 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:55:58.165187 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:58.165206 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:58.165707 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:58.166082 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:58.166649 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:58.167923 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:58.169271 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:58.170007 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:58.170209 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:58.171352 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:58.171359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:58.171397 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:55:58.171892 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:58.171964 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:55:58.171989 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:58.172112 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:58.172324 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 22:55:58.172615 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":345,"threads_started":1}
I20260927 22:55:58.176611 18123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:58.177546 18123 log.cc:824] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:58.180195 18123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be: No bootstrap required, opened a new log
I20260927 22:55:58.187129 18123 raft_consensus.cc:348] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48cef7754024b03ae1bc2a81242b8be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40455 } }
I20260927 22:55:58.187465 18123 raft_consensus.cc:374] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:58.187587 18123 raft_consensus.cc:729] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48cef7754024b03ae1bc2a81242b8be, State: Initialized, Role: FOLLOWER
I20260927 22:55:58.188170 18123 consensus_queue.cc:260] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48cef7754024b03ae1bc2a81242b8be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40455 } }
I20260927 22:55:58.188400 18123 raft_consensus.cc:388] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:55:58.188555 18123 raft_consensus.cc:482] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:55:58.188719 18123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:58.190855 18123 raft_consensus.cc:504] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48cef7754024b03ae1bc2a81242b8be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40455 } }
I20260927 22:55:58.191339 18123 leader_election.cc:302] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e48cef7754024b03ae1bc2a81242b8be; no voters:
I20260927 22:55:58.191906 18123 leader_election.cc:288] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:55:58.192044 18128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:55:58.192730 18128 raft_consensus.cc:686] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [term 1 LEADER]: Becoming Leader. State: Replica: e48cef7754024b03ae1bc2a81242b8be, State: Running, Role: LEADER
I20260927 22:55:58.193315 18128 consensus_queue.cc:237] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48cef7754024b03ae1bc2a81242b8be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40455 } }
I20260927 22:55:58.194048 18123 sys_catalog.cc:564] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:55:58.196496 18130 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [sys.catalog]: SysCatalogTable state changed. Reason: New leader e48cef7754024b03ae1bc2a81242b8be. Latest consensus state: current_term: 1 leader_uuid: "e48cef7754024b03ae1bc2a81242b8be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48cef7754024b03ae1bc2a81242b8be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40455 } } }
I20260927 22:55:58.197113 18130 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [sys.catalog]: This master's current role is: LEADER
I20260927 22:55:58.196533 18129 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e48cef7754024b03ae1bc2a81242b8be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48cef7754024b03ae1bc2a81242b8be" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40455 } } }
I20260927 22:55:58.197573 18129 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be [sys.catalog]: This master's current role is: LEADER
I20260927 22:55:58.198596 18138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:55:58.203122 18138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:55:58.210757 18138 catalog_manager.cc:1347] Generated new cluster ID: 6e2b6750662b45f6967558cadfb7bf74
I20260927 22:55:58.210863 18138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:55:58.236621 18138 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:55:58.237943 18138 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:55:58.242592 18138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be: Generated new TSK 0
I20260927 22:55:58.243341 18138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:55:58.254640 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40455
--builtin_ntp_servers=127.0.97.20:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:55:58.577929 18147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:58.578229 18147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:58.578388 18147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:58.585459 18147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:58.585585 18147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:58.585783 18147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:55:58.594781 18147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:58.596009 18147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:58.597824 18147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:58.610929 18155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:58.611091 18153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:58.610976 18152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:58.611650 18147 server_base.cc:1034] running on GCE node
I20260927 22:55:58.612151 18147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:58.612967 18147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:58.614152 18147 hybrid_clock.cc:648] HybridClock initialized: now 1790549758614097 us; error 52 us; skew 500 ppm
I20260927 22:55:58.614408 18147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:55:58.616427 18147 webserver.cc:460] Webserver started at http://127.0.97.1:41079/ using document root <none> and password file <none>
I20260927 22:55:58.617015 18147 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:58.617096 18147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:58.617340 18147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:55:58.620013 18147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "2bbe591154094e628aa577298a98ede0"
format_stamp: "Formatted at 2026-09-27 22:55:58 on dist-test-slave-df53"
server_key: "b6ab59ccf1a4191dd26f143494392cea"
server_key_iv: "cf5f3749c730ec8ad4871dca17d3b72b"
server_key_version: "encryptionkey@0"
I20260927 22:55:58.620702 18147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2bbe591154094e628aa577298a98ede0"
format_stamp: "Formatted at 2026-09-27 22:55:58 on dist-test-slave-df53"
server_key: "b6ab59ccf1a4191dd26f143494392cea"
server_key_iv: "cf5f3749c730ec8ad4871dca17d3b72b"
server_key_version: "encryptionkey@0"
I20260927 22:55:58.625252 18147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 22:55:58.628468 18161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:58.629810 18147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:55:58.629973 18147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "2bbe591154094e628aa577298a98ede0"
format_stamp: "Formatted at 2026-09-27 22:55:58 on dist-test-slave-df53"
server_key: "b6ab59ccf1a4191dd26f143494392cea"
server_key_iv: "cf5f3749c730ec8ad4871dca17d3b72b"
server_key_version: "encryptionkey@0"
I20260927 22:55:58.630177 18147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:58.663271 18147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:58.664045 18147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:58.664316 18147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:58.665284 18147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:58.666630 18147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:58.666751 18147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:58.666877 18147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:58.666961 18147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:58.699707 18147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39967
I20260927 22:55:58.699810 18273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39967 every 8 connection(s)
I20260927 22:55:58.700976 18147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:55:58.709856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 18147
I20260927 22:55:58.710032 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:55:58.710461 388 external_mini_cluster.cc:1428] Setting key 9c8173e6db8e3337f8453e1ebe1306c0
I20260927 22:55:58.713297 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40455
--builtin_ntp_servers=127.0.97.20:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:55:58.713503 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:58.703551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35795 (local address 127.0.97.62:40455)
0927 22:55:58.704198 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:58.704206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:58.704937 (+ 731us) server_negotiation.cc:407] Connection header received
0927 22:55:58.706977 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:58.706983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:58.707067 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:58.707192 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:58.708207 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:58.708944 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:58.710641 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:58.711181 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:58.711301 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:58.712698 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:58.712703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:58.712706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:55:58.712722 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:58.712757 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:55:58.712769 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:58.712839 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:58.713109 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:55:58.713183 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":242,"threads_started":1}
I20260927 22:55:58.714960 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:58.703972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40455 (local address 127.0.97.1:35795)
0927 22:55:58.704795 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:58.704830 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:58.706712 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:58.707414 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:58.707422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:58.707505 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:58.708048 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:58.708064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:58.709101 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:58.709105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:58.710472 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:58.710489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:58.711412 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:58.711420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:58.711622 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:58.712359 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:55:58.712363 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:58.712410 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:55:58.712499 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:58.712869 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:58.712880 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:58.712889 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:55:58.713087 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 22:55:58.713469 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":257,"threads_started":1}
I20260927 22:55:58.716384 18274 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40455
I20260927 22:55:58.716661 18274 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:58.717420 18274 heartbeater.cc:511] Master 127.0.97.62:40455 requested a full tablet report, sending...
I20260927 22:55:58.719388 18088 ts_manager.cc:194] Registered new tserver with Master: 2bbe591154094e628aa577298a98ede0 (127.0.97.1:39967)
I20260927 22:55:58.720849 18088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35795
W20260927 22:55:59.024166 18278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:59.024425 18278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:59.024547 18278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:59.031414 18278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:59.031498 18278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:59.031596 18278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:55:59.040380 18278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:59.041478 18278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:59.043094 18278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:59.055538 18284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:59.055693 18283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:59.056255 18286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:59.056249 18278 server_base.cc:1034] running on GCE node
I20260927 22:55:59.056913 18278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:59.057823 18278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:59.059062 18278 hybrid_clock.cc:648] HybridClock initialized: now 1790549759059004 us; error 38 us; skew 500 ppm
I20260927 22:55:59.059355 18278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:55:59.061436 18278 webserver.cc:460] Webserver started at http://127.0.97.2:32773/ using document root <none> and password file <none>
I20260927 22:55:59.062112 18278 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:59.062196 18278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:59.062450 18278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:55:59.065110 18278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "9381a17c69b44f4b8217dad229c8f8c2"
format_stamp: "Formatted at 2026-09-27 22:55:59 on dist-test-slave-df53"
server_key: "f371f0cf291b8c659ab9346fe1d0a797"
server_key_iv: "f50cb3446124fa2b47c099d04e885738"
server_key_version: "encryptionkey@0"
I20260927 22:55:59.065876 18278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9381a17c69b44f4b8217dad229c8f8c2"
format_stamp: "Formatted at 2026-09-27 22:55:59 on dist-test-slave-df53"
server_key: "f371f0cf291b8c659ab9346fe1d0a797"
server_key_iv: "f50cb3446124fa2b47c099d04e885738"
server_key_version: "encryptionkey@0"
I20260927 22:55:59.070504 18278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:55:59.073854 18292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:59.075057 18278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:55:59.075217 18278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "9381a17c69b44f4b8217dad229c8f8c2"
format_stamp: "Formatted at 2026-09-27 22:55:59 on dist-test-slave-df53"
server_key: "f371f0cf291b8c659ab9346fe1d0a797"
server_key_iv: "f50cb3446124fa2b47c099d04e885738"
server_key_version: "encryptionkey@0"
I20260927 22:55:59.075423 18278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:59.092692 18278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:59.093459 18278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:59.093777 18278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:59.094807 18278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:59.096297 18278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:59.096418 18278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:59.096544 18278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:59.096638 18278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:59.128475 18278 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44217
I20260927 22:55:59.128559 18404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44217 every 8 connection(s)
I20260927 22:55:59.129840 18278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:55:59.136353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 18278
I20260927 22:55:59.136528 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:55:59.137094 388 external_mini_cluster.cc:1428] Setting key d95bdae50331a64fb0931e45cbfa8dbd
I20260927 22:55:59.139767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40455
--builtin_ntp_servers=127.0.97.20:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:55:59.142400 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.132263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49545 (local address 127.0.97.62:40455)
0927 22:55:59.132434 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:59.132438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:59.133804 (+ 1366us) server_negotiation.cc:407] Connection header received
0927 22:55:59.135514 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:59.135519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:59.135589 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:59.135677 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:59.136770 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.137842 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.139565 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.140057 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.140187 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.141577 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:59.141582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:59.141625 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:55:59.141647 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:59.141686 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:55:59.141699 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:59.141790 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:59.141999 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 22:55:59.142092 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:55:59.143318 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.132697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40455 (local address 127.0.97.2:49545)
0927 22:55:59.133663 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:59.133697 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:59.135276 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:59.135873 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:59.135880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:59.135973 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:59.136556 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.136596 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.138011 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.138016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.139412 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.139424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.140277 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.140284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.140515 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.141254 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:55:59.141256 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:59.141292 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:55:59.141366 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:59.141789 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:59.141800 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:59.141810 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:55:59.141987 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 22:55:59.142339 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":280,"threads_started":1}
I20260927 22:55:59.144702 18405 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40455
I20260927 22:55:59.144977 18405 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:59.145709 18405 heartbeater.cc:511] Master 127.0.97.62:40455 requested a full tablet report, sending...
I20260927 22:55:59.147061 18088 ts_manager.cc:194] Registered new tserver with Master: 9381a17c69b44f4b8217dad229c8f8c2 (127.0.97.2:44217)
I20260927 22:55:59.147848 18088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:49545
W20260927 22:55:59.453541 18408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:59.453893 18408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:59.454049 18408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:59.461427 18408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:59.461560 18408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:59.461755 18408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:55:59.470777 18408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:55:59.471998 18408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:59.473692 18408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:59.486142 18414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:59.486225 18413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:59.486225 18416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:59.487401 18408 server_base.cc:1034] running on GCE node
I20260927 22:55:59.487963 18408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:59.488850 18408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:59.490072 18408 hybrid_clock.cc:648] HybridClock initialized: now 1790549759490032 us; error 51 us; skew 500 ppm
I20260927 22:55:59.490373 18408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:55:59.498857 18408 webserver.cc:460] Webserver started at http://127.0.97.3:32775/ using document root <none> and password file <none>
I20260927 22:55:59.499536 18408 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:59.499653 18408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:59.500022 18408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:55:59.502877 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "ce14298a43b34fb1b7627025625fd1af"
format_stamp: "Formatted at 2026-09-27 22:55:59 on dist-test-slave-df53"
server_key: "566d6fea13ffffbe835e3cbac8dedd5c"
server_key_iv: "7ea4026ce30fa139ddcdfac3d7998ecf"
server_key_version: "encryptionkey@0"
I20260927 22:55:59.503684 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ce14298a43b34fb1b7627025625fd1af"
format_stamp: "Formatted at 2026-09-27 22:55:59 on dist-test-slave-df53"
server_key: "566d6fea13ffffbe835e3cbac8dedd5c"
server_key_iv: "7ea4026ce30fa139ddcdfac3d7998ecf"
server_key_version: "encryptionkey@0"
I20260927 22:55:59.509238 18408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:55:59.512610 18422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:59.513911 18408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:55:59.514076 18408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "ce14298a43b34fb1b7627025625fd1af"
format_stamp: "Formatted at 2026-09-27 22:55:59 on dist-test-slave-df53"
server_key: "566d6fea13ffffbe835e3cbac8dedd5c"
server_key_iv: "7ea4026ce30fa139ddcdfac3d7998ecf"
server_key_version: "encryptionkey@0"
I20260927 22:55:59.514279 18408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:59.530476 18408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:59.531185 18408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:59.531481 18408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:59.532379 18408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:59.533980 18408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:59.534099 18408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:59.534216 18408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:59.534298 18408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:59.567529 18408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38385
I20260927 22:55:59.567575 18534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38385 every 8 connection(s)
I20260927 22:55:59.568722 18408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:55:59.573244 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 18408
I20260927 22:55:59.573575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:55:59.574595 388 external_mini_cluster.cc:1428] Setting key 7c4745c039d5d594a9741690e2f4f776
I20260927 22:55:59.584596 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.571331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33469 (local address 127.0.97.62:40455)
0927 22:55:59.571577 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:59.571583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:59.574602 (+ 3019us) server_negotiation.cc:407] Connection header received
0927 22:55:59.577228 (+ 2626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:59.577234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:59.577321 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:59.577443 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:59.579080 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.580207 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.582245 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.582685 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.582782 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.583949 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:59.583954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:59.583957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:55:59.583974 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:59.584003 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:55:59.584014 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:59.584078 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:59.584273 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 22:55:59.584344 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 22:55:59.585534 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.571908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40455 (local address 127.0.97.3:33469)
0927 22:55:59.574440 (+ 2532us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:59.574483 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:59.576904 (+ 2421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:59.577941 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:59.577954 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:59.578065 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:59.578858 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.578887 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.580440 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.580445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.582116 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.582128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.582862 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.582867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.583065 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.583661 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:55:59.583663 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:59.583705 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:55:59.583774 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:59.584087 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:59.584092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:59.584097 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:55:59.584232 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:55:59.584495 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1999,"thread_start_us":1830,"threads_started":1}
I20260927 22:55:59.586822 18535 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40455
I20260927 22:55:59.587110 18535 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:59.587805 18535 heartbeater.cc:511] Master 127.0.97.62:40455 requested a full tablet report, sending...
I20260927 22:55:59.589213 18088 ts_manager.cc:194] Registered new tserver with Master: ce14298a43b34fb1b7627025625fd1af (127.0.97.3:38385)
I20260927 22:55:59.590339 18088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:33469
I20260927 22:55:59.590696 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:55:59.600360 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.594588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.0.97.62:40455)
0927 22:55:59.594817 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:59.594823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:59.595158 (+ 335us) server_negotiation.cc:407] Connection header received
0927 22:55:59.595341 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:59.595346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:59.595422 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:59.595543 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:59.596016 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.596672 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.598111 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.598510 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.598642 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.599703 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:59.599709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:59.599715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:55:59.599736 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:59.599764 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:55:59.599775 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:59.599867 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:59.600007 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 22:55:59.600090 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:55:59.603571 18088 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:55:59.610522 18088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:55:59.633380 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.621981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44217 (local address 127.0.0.1:42652)
0927 22:55:59.622690 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:59.622727 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:59.622900 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:59.623957 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:59.623967 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:59.624009 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:59.624477 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.624492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.627018 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.627025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.629673 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.629687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.632172 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.632181 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.632466 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.632535 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 22:55:59.632681 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 22:55:59.632932 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":321,"threads_started":1}
I20260927 22:55:59.634771 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.622270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.0.97.2:44217)
0927 22:55:59.622777 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:59.622788 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:59.622837 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:55:59.623169 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:59.623180 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:59.623455 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:59.623636 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:59.624739 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.626804 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.629886 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.631510 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.631674 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.632025 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:59.634256 (+ 2231us) server_negotiation.cc:299] Negotiation successful
0927 22:55:59.634416 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":803,"threads_started":1}
I20260927 22:55:59.638762 18340 tablet_service.cc:1467] Processing CreateTablet for tablet 507fec35a82c48b5aefc0ed8904ab2a9 (DEFAULT_TABLE table=test-table [id=b4dca8d3c0b945848d17a4e48b6dd858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:59.640118 18340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507fec35a82c48b5aefc0ed8904ab2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:59.641549 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.623678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38385 (local address 127.0.0.1:54134)
0927 22:55:59.633970 (+ 10292us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:59.633992 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:59.634097 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:59.635337 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:59.635341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:59.635368 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:59.635525 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.635535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.637314 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.637321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.639089 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.639100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.640760 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.640766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.640940 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.640979 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:55:59.641074 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:55:59.641206 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10120,"mutex_wait_us":2,"thread_start_us":11152,"threads_started":1}
I20260927 22:55:59.641665 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.624200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.0.97.3:38385)
0927 22:55:59.634386 (+ 10186us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:59.634394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:59.634426 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:55:59.634521 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:59.634528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:59.634772 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:59.635360 (+ 588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:59.635688 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.637166 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.639247 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.640588 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.640728 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.640988 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:59.641115 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 22:55:59.641290 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10027,"thread_start_us":9901,"threads_started":1}
I20260927 22:55:59.642204 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.623084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39967 (local address 127.0.0.1:48760)
0927 22:55:59.623667 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:59.623689 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:59.623828 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:59.635554 (+ 11726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:59.635559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:59.635583 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:59.635977 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.635990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.637417 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.637422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.639533 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.639547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.641377 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.641386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.641621 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.641664 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:55:59.641735 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:55:59.641875 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":289,"threads_started":1}
I20260927 22:55:59.643486 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.623367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.0.97.1:39967)
0927 22:55:59.634980 (+ 11613us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:59.634987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:59.635009 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:55:59.635115 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:59.635124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:59.635350 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:59.636146 (+ 796us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:59.636299 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.637274 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.639728 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.641150 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.642704 (+ 1554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.642865 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:59.642962 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 22:55:59.643140 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11447,"thread_start_us":11328,"threads_started":1}
I20260927 22:55:59.645313 18470 tablet_service.cc:1467] Processing CreateTablet for tablet 507fec35a82c48b5aefc0ed8904ab2a9 (DEFAULT_TABLE table=test-table [id=b4dca8d3c0b945848d17a4e48b6dd858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:59.646505 18470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507fec35a82c48b5aefc0ed8904ab2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:59.646809 18209 tablet_service.cc:1467] Processing CreateTablet for tablet 507fec35a82c48b5aefc0ed8904ab2a9 (DEFAULT_TABLE table=test-table [id=b4dca8d3c0b945848d17a4e48b6dd858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:59.648022 18209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507fec35a82c48b5aefc0ed8904ab2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:59.654104 18553 tablet_bootstrap.cc:492] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: Bootstrap starting.
I20260927 22:55:59.657611 18553 tablet_bootstrap.cc:654] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:59.658690 18554 tablet_bootstrap.cc:492] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: Bootstrap starting.
I20260927 22:55:59.658941 18553 log.cc:824] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:59.661356 18553 tablet_bootstrap.cc:492] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: No bootstrap required, opened a new log
I20260927 22:55:59.661680 18553 ts_tablet_manager.cc:1397] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260927 22:55:59.662124 18554 tablet_bootstrap.cc:654] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:59.662307 18556 tablet_bootstrap.cc:492] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0: Bootstrap starting.
I20260927 22:55:59.663250 18554 log.cc:824] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:59.665674 18556 tablet_bootstrap.cc:654] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:59.666276 18554 tablet_bootstrap.cc:492] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: No bootstrap required, opened a new log
I20260927 22:55:59.666536 18554 ts_tablet_manager.cc:1397] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:55:59.666849 18556 log.cc:824] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:59.669553 18556 tablet_bootstrap.cc:492] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0: No bootstrap required, opened a new log
I20260927 22:55:59.669847 18556 ts_tablet_manager.cc:1397] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 22:55:59.671715 18553 raft_consensus.cc:348] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9381a17c69b44f4b8217dad229c8f8c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44217 } } peers { permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 } } peers { permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 } }
I20260927 22:55:59.672137 18553 raft_consensus.cc:374] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:59.672269 18553 raft_consensus.cc:729] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9381a17c69b44f4b8217dad229c8f8c2, State: Initialized, Role: FOLLOWER
I20260927 22:55:59.672866 18553 consensus_queue.cc:260] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9381a17c69b44f4b8217dad229c8f8c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44217 } } peers { permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 } } peers { permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 } }
I20260927 22:55:59.673996 18405 heartbeater.cc:503] Master 127.0.97.62:40455 was elected leader, sending a full tablet report...
I20260927 22:55:59.674014 18554 raft_consensus.cc:348] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9381a17c69b44f4b8217dad229c8f8c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44217 } } peers { permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 } } peers { permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 } }
I20260927 22:55:59.674389 18554 raft_consensus.cc:374] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:59.674479 18554 raft_consensus.cc:729] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce14298a43b34fb1b7627025625fd1af, State: Initialized, Role: FOLLOWER
I20260927 22:55:59.674934 18553 ts_tablet_manager.cc:1428] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260927 22:55:59.675007 18554 consensus_queue.cc:260] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9381a17c69b44f4b8217dad229c8f8c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44217 } } peers { permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 } } peers { permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 } }
I20260927 22:55:59.675963 18535 heartbeater.cc:503] Master 127.0.97.62:40455 was elected leader, sending a full tablet report...
I20260927 22:55:59.678213 18554 ts_tablet_manager.cc:1428] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260927 22:55:59.679891 18556 raft_consensus.cc:348] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9381a17c69b44f4b8217dad229c8f8c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44217 } } peers { permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 } } peers { permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 } }
I20260927 22:55:59.680368 18556 raft_consensus.cc:374] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:59.680505 18556 raft_consensus.cc:729] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbe591154094e628aa577298a98ede0, State: Initialized, Role: FOLLOWER
I20260927 22:55:59.681090 18556 consensus_queue.cc:260] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9381a17c69b44f4b8217dad229c8f8c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44217 } } peers { permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 } } peers { permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 } }
I20260927 22:55:59.682008 18274 heartbeater.cc:503] Master 127.0.97.62:40455 was elected leader, sending a full tablet report...
I20260927 22:55:59.682555 18556 ts_tablet_manager.cc:1428] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
W20260927 22:55:59.703189 18275 tablet.cc:2369] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:55:59.757499 18559 raft_consensus.cc:482] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:55:59.757839 18559 raft_consensus.cc:504] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9381a17c69b44f4b8217dad229c8f8c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44217 } } peers { permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 } } peers { permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 } }
I20260927 22:55:59.759452 18559 leader_election.cc:288] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bbe591154094e628aa577298a98ede0 (127.0.97.1:39967), ce14298a43b34fb1b7627025625fd1af (127.0.97.3:38385)
I20260927 22:55:59.765834 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.759847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54843 (local address 127.0.97.1:39967)
0927 22:55:59.760049 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:59.760054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:59.760613 (+ 559us) server_negotiation.cc:407] Connection header received
0927 22:55:59.760826 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:59.760832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:59.760932 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:59.761040 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:59.761765 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.762444 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.764189 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.764935 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.765084 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.765187 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:59.765451 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 22:55:59.765572 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:55:59.765944 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.759811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39967 (local address 127.0.97.2:54843)
0927 22:55:59.760496 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:59.760517 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:59.760659 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:59.761211 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:59.761214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:59.761236 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:59.761567 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.761575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.762578 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.762585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.764003 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.764014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.765126 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.765132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.765294 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.765317 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:55:59.765423 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:55:59.765545 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":2,"thread_start_us":364,"threads_started":1}
I20260927 22:55:59.766433 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.760933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55353 (local address 127.0.97.3:38385)
0927 22:55:59.761116 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:59.761121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:59.761419 (+ 298us) server_negotiation.cc:407] Connection header received
0927 22:55:59.761638 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:59.761644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:59.761710 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:59.761804 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:59.762338 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.763021 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.764711 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.765465 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.765627 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.765724 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:59.766070 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 22:55:59.766187 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:55:59.766929 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.760611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38385 (local address 127.0.97.2:55353)
0927 22:55:59.761294 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:59.761312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:59.761463 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:59.761822 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:59.761825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:59.761847 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:59.762186 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.762194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.763167 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.763172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.764527 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:59.764538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.765673 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.765679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:59.765839 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.765866 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 22:55:59.766599 (+ 733us) client_negotiation.cc:240] Negotiation successful
0927 22:55:59.766727 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":390,"threads_started":1}
I20260927 22:55:59.767273 18229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507fec35a82c48b5aefc0ed8904ab2a9" candidate_uuid: "9381a17c69b44f4b8217dad229c8f8c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bbe591154094e628aa577298a98ede0" is_pre_election: true
I20260927 22:55:59.767763 18229 raft_consensus.cc:2454] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9381a17c69b44f4b8217dad229c8f8c2 in term 0.
I20260927 22:55:59.767855 18490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507fec35a82c48b5aefc0ed8904ab2a9" candidate_uuid: "9381a17c69b44f4b8217dad229c8f8c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce14298a43b34fb1b7627025625fd1af" is_pre_election: true
I20260927 22:55:59.768378 18490 raft_consensus.cc:2454] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9381a17c69b44f4b8217dad229c8f8c2 in term 0.
I20260927 22:55:59.768396 18294 leader_election.cc:302] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe591154094e628aa577298a98ede0, 9381a17c69b44f4b8217dad229c8f8c2; no voters:
I20260927 22:55:59.768766 18559 raft_consensus.cc:2781] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:55:59.768915 18559 raft_consensus.cc:482] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:55:59.769026 18559 raft_consensus.cc:3037] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:59.771243 18559 raft_consensus.cc:504] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9381a17c69b44f4b8217dad229c8f8c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44217 } } peers { permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 } } peers { permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 } }
I20260927 22:55:59.771811 18559 leader_election.cc:288] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [CANDIDATE]: Term 1 election: Requested vote from peers 2bbe591154094e628aa577298a98ede0 (127.0.97.1:39967), ce14298a43b34fb1b7627025625fd1af (127.0.97.3:38385)
I20260927 22:55:59.772219 18229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507fec35a82c48b5aefc0ed8904ab2a9" candidate_uuid: "9381a17c69b44f4b8217dad229c8f8c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bbe591154094e628aa577298a98ede0"
I20260927 22:55:59.772369 18490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507fec35a82c48b5aefc0ed8904ab2a9" candidate_uuid: "9381a17c69b44f4b8217dad229c8f8c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce14298a43b34fb1b7627025625fd1af"
I20260927 22:55:59.772421 18229 raft_consensus.cc:3037] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:59.772570 18490 raft_consensus.cc:3037] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:59.774711 18229 raft_consensus.cc:2454] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9381a17c69b44f4b8217dad229c8f8c2 in term 1.
I20260927 22:55:59.774812 18490 raft_consensus.cc:2454] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9381a17c69b44f4b8217dad229c8f8c2 in term 1.
I20260927 22:55:59.775123 18294 leader_election.cc:302] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe591154094e628aa577298a98ede0, 9381a17c69b44f4b8217dad229c8f8c2; no voters:
I20260927 22:55:59.775418 18559 raft_consensus.cc:2781] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:55:59.775830 18559 raft_consensus.cc:686] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 1 LEADER]: Becoming Leader. State: Replica: 9381a17c69b44f4b8217dad229c8f8c2, State: Running, Role: LEADER
I20260927 22:55:59.776279 18559 consensus_queue.cc:237] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9381a17c69b44f4b8217dad229c8f8c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44217 } } peers { permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 } } peers { permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 } }
I20260927 22:55:59.780895 18087 catalog_manager.cc:5495] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9381a17c69b44f4b8217dad229c8f8c2 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9381a17c69b44f4b8217dad229c8f8c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9381a17c69b44f4b8217dad229c8f8c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 } health_report { overall_health: UNKNOWN } } }
W20260927 22:55:59.820263 18536 tablet.cc:2369] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:55:59.828104 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.823145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42662 (local address 127.0.97.2:44217)
0927 22:55:59.823349 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:59.823357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:59.823415 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:55:59.823606 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:59.823609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:59.823691 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:59.823808 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:55:59.824096 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.824756 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.826192 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.826605 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.826700 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.827121 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:55:59.827577 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:55:59.827688 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:59.827756 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 22:55:59.827813 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:55:59.835807 18229 raft_consensus.cc:1261] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0 [term 1 FOLLOWER]: Refusing update from remote peer 9381a17c69b44f4b8217dad229c8f8c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:55:59.835899 18490 raft_consensus.cc:1261] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af [term 1 FOLLOWER]: Refusing update from remote peer 9381a17c69b44f4b8217dad229c8f8c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:55:59.836786 18564 consensus_queue.cc:1035] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bbe591154094e628aa577298a98ede0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:55:59.837128 18559 consensus_queue.cc:1035] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce14298a43b34fb1b7627025625fd1af" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:55:59.848632 18568 mvcc.cc:205] Tried to move back new op lower bound from 7334091816278122496 to 7334091816047386624. Current Snapshot: MvccSnapshot[applied={T|T < 7334091816278122496}]
I20260927 22:55:59.862145 18087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:55:59.862426 18087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:55:59.865736 18087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e48cef7754024b03ae1bc2a81242b8be: Sending DeleteTablet for 3 replicas of tablet 507fec35a82c48b5aefc0ed8904ab2a9
I20260927 22:55:59.866796 18340 tablet_service.cc:1514] Processing DeleteTablet for tablet 507fec35a82c48b5aefc0ed8904ab2a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:59 UTC) from {username='slave'} at 127.0.0.1:42652
I20260927 22:55:59.867005 18470 tablet_service.cc:1514] Processing DeleteTablet for tablet 507fec35a82c48b5aefc0ed8904ab2a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:59 UTC) from {username='slave'} at 127.0.0.1:54134
I20260927 22:55:59.867117 18209 tablet_service.cc:1514] Processing DeleteTablet for tablet 507fec35a82c48b5aefc0ed8904ab2a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:59 UTC) from {username='slave'} at 127.0.0.1:48760
I20260927 22:55:59.868347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 18147
I20260927 22:55:59.885315 18577 tablet_replica.cc:330] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: stopping tablet replica
I20260927 22:55:59.886431 18578 tablet_replica.cc:330] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: stopping tablet replica
I20260927 22:55:59.886597 18577 raft_consensus.cc:2229] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:55:59.886906 18578 raft_consensus.cc:2229] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:55:59.887416 18577 raft_consensus.cc:2258] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:59.887444 18578 raft_consensus.cc:2258] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:59.887984 18579 tablet_replica.cc:330] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0: stopping tablet replica
I20260927 22:55:59.888435 18579 raft_consensus.cc:2229] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:55:59.888969 18579 raft_consensus.cc:2258] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:59.890002 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:59.869686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.0.97.1:39967)
0927 22:55:59.869874 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:59.869880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:59.876535 (+ 6655us) server_negotiation.cc:407] Connection header received
0927 22:55:59.877531 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:59.877540 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:59.877685 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:59.877770 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:59.879182 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.880755 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.885953 (+ 5198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:59.886607 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:59.886746 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:59.888300 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:59.888312 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:59.888329 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:55:59.889131 (+ 802us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:59.889194 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:55:59.889218 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:59.889339 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:59.889528 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 22:55:59.889666 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:55:59.890096 18578 ts_tablet_manager.cc:1905] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:59.890796 18269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:59.892891 18577 ts_tablet_manager.cc:1905] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:59.900403 18577 ts_tablet_manager.cc:1918] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:59.900629 18577 log.cc:1196] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/507fec35a82c48b5aefc0ed8904ab2a9
I20260927 22:55:59.901243 18577 ts_tablet_manager.cc:1939] T 507fec35a82c48b5aefc0ed8904ab2a9 P 9381a17c69b44f4b8217dad229c8f8c2: Deleting consensus metadata
I20260927 22:55:59.901731 18578 ts_tablet_manager.cc:1918] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:59.901947 18578 log.cc:1196] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/507fec35a82c48b5aefc0ed8904ab2a9
I20260927 22:55:59.902488 18578 ts_tablet_manager.cc:1939] T 507fec35a82c48b5aefc0ed8904ab2a9 P ce14298a43b34fb1b7627025625fd1af: Deleting consensus metadata
I20260927 22:55:59.903146 18073 catalog_manager.cc:4841] TS 9381a17c69b44f4b8217dad229c8f8c2 (127.0.97.2:44217): tablet 507fec35a82c48b5aefc0ed8904ab2a9 (table test-table [id=b4dca8d3c0b945848d17a4e48b6dd858]) successfully deleted
I20260927 22:55:59.904160 18073 catalog_manager.cc:4841] TS ce14298a43b34fb1b7627025625fd1af (127.0.97.3:38385): tablet 507fec35a82c48b5aefc0ed8904ab2a9 (table test-table [id=b4dca8d3c0b945848d17a4e48b6dd858]) successfully deleted
I20260927 22:55:59.891664 18579 ts_tablet_manager.cc:1905] T 507fec35a82c48b5aefc0ed8904ab2a9 P 2bbe591154094e628aa577298a98ede0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:59.985435 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 18147
I20260927 22:56:00.003587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 18278
W20260927 22:56:00.003615 18073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39967 (error 108)
I20260927 22:56:00.006364 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:00.005844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39967 (local address 127.0.0.1:48782)
0927 22:56:00.006067 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:00.006193 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 22:56:00.006626 18073 catalog_manager.cc:4584] TS 2bbe591154094e628aa577298a98ede0 (127.0.97.1:39967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 507fec35a82c48b5aefc0ed8904ab2a9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39967: connect: Connection refused (error 111)
I20260927 22:56:00.011339 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:00.004817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.0.97.2:44217)
0927 22:56:00.005041 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:00.005047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:00.005065 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:56:00.005128 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:00.005134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:00.005238 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:00.005344 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:00.005903 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:00.007061 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:00.008315 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:00.008725 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:00.008883 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:00.009978 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:00.009986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:00.010000 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:00.010580 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:00.010628 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:00.010647 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:00.010782 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:00.010913 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:56:00.011014 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:56:00.012013 18400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:00.060220 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:00.059671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39967 (local address 127.0.0.1:48786)
0927 22:56:00.059919 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:00.060027 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 22:56:00.061261 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:00.060876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39967 (local address 127.0.0.1:48790)
0927 22:56:00.061038 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:00.061120 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:56:00.121984 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:00.121420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39967 (local address 127.0.0.1:48806)
0927 22:56:00.121667 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:00.121770 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:56:00.122014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 18278
I20260927 22:56:00.124711 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:00.124358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39967 (local address 127.0.0.1:48808)
0927 22:56:00.124510 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:00.124594 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:56:00.139987 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 18408
I20260927 22:56:00.146859 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:00.140825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54146 (local address 127.0.97.3:38385)
0927 22:56:00.141015 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:00.141021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:00.141121 (+ 100us) server_negotiation.cc:407] Connection header received
0927 22:56:00.141336 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:00.141340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:00.141429 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:00.141534 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:00.141962 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:00.142720 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:00.144008 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:00.144401 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:00.144533 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:00.145524 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:00.145533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:00.145547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:00.146146 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:00.146198 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:00.146217 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:00.146342 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:00.146472 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 22:56:00.146577 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:56:00.147441 18530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:00.205000 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:00.204573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39967 (local address 127.0.0.1:48812)
0927 22:56:00.204755 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:00.204837 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:56:00.206035 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:00.205664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39967 (local address 127.0.0.1:48814)
0927 22:56:00.205854 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:00.205930 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 22:56:00.234100 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 18408
I20260927 22:56:00.250025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 18057
I20260927 22:56:00.251099 18118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:00.362718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 18057
2026-09-27T22:56:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2703 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:00 dist-test-slave-df53 krb5kdc[18588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:00 dist-test-slave-df53 krb5kdc[18588](info): set up 2 sockets
Sep 27 22:56:00 dist-test-slave-df53 krb5kdc[18588](info): commencing operation
krb5kdc: starting...
W20260927 22:56:01.461686 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:01 dist-test-slave-df53 krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:56:01.688812 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39545
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:02.004758 18601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:02.005028 18601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:02.005170 18601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:02.012439 18601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:02.012538 18601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:02.012640 18601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:02.012741 18601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:02.012804 18601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:02.022338 18601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:02.023541 18601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:02.024799 18601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:02.036307 18607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:02.036310 18609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:02.036358 18606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:02.037092 18601 server_base.cc:1034] running on GCE node
I20260927 22:56:02.037863 18601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:02.039099 18601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:02.040292 18601 hybrid_clock.cc:648] HybridClock initialized: now 1790549762040232 us; error 48 us; skew 500 ppm
I20260927 22:56:02.040616 18601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:02 dist-test-slave-df53 krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:02.043761 18601 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:56:02.045228 18601 webserver.cc:460] Webserver started at http://127.0.97.62:34179/ using document root <none> and password file <none>
I20260927 22:56:02.045956 18601 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:02.046084 18601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:02.046406 18601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:02.049192 18601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "d2a3313cf3dc474290fb4eb2f371469a"
format_stamp: "Formatted at 2026-09-27 22:56:02 on dist-test-slave-df53"
server_key: "4d73ed6449b8b2cec046fba8d820e802"
server_key_iv: "3564458526ad8d0350e4d4b4595f0397"
server_key_version: "encryptionkey@0"
I20260927 22:56:02.049952 18601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "d2a3313cf3dc474290fb4eb2f371469a"
format_stamp: "Formatted at 2026-09-27 22:56:02 on dist-test-slave-df53"
server_key: "4d73ed6449b8b2cec046fba8d820e802"
server_key_iv: "3564458526ad8d0350e4d4b4595f0397"
server_key_version: "encryptionkey@0"
I20260927 22:56:02.054857 18601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:56:02.058140 18616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:02.059453 18601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:56:02.059619 18601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "d2a3313cf3dc474290fb4eb2f371469a"
format_stamp: "Formatted at 2026-09-27 22:56:02 on dist-test-slave-df53"
server_key: "4d73ed6449b8b2cec046fba8d820e802"
server_key_iv: "3564458526ad8d0350e4d4b4595f0397"
server_key_version: "encryptionkey@0"
I20260927 22:56:02.059798 18601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:02.078068 18601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:02.081969 18601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:02.082240 18601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:02.100378 18601 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39545
I20260927 22:56:02.100420 18667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39545 every 8 connection(s)
I20260927 22:56:02.101684 18601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:02.105134 18668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:02.109829 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 18601
I20260927 22:56:02.110006 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:02.110491 388 external_mini_cluster.cc:1428] Setting key 6759c74e639298e4ea6cd182f20ac228
I20260927 22:56:02.115816 18668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a: Bootstrap starting.
I20260927 22:56:02.119266 18668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:02.120457 18668 log.cc:824] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:02.123175 18668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a: No bootstrap required, opened a new log
Sep 27 22:56:02 dist-test-slave-df53 krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:02.130853 18668 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a3313cf3dc474290fb4eb2f371469a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39545 } }
I20260927 22:56:02.131217 18668 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:02.131393 18668 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a3313cf3dc474290fb4eb2f371469a, State: Initialized, Role: FOLLOWER
I20260927 22:56:02.131948 18668 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a3313cf3dc474290fb4eb2f371469a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39545 } }
I20260927 22:56:02.132203 18668 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:02.132390 18668 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:02.132584 18668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:02.132907 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:02.112193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.0.97.62:39545)
0927 22:56:02.112997 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:02.113007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:02.113050 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:56:02.114803 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:02.114824 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:02.115162 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:02.115470 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:02.117154 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:02.118518 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:02.120407 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:02.121112 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:02.121325 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:02.125384 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:02.125391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:02.125410 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:02.125450 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:02.128493 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:02.129183 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:02.129189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:02.129194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:02.129357 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:02.129835 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:02.129839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:02.129841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:02.130338 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:02.130632 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:02.131087 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 22:56:02.131723 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":378,"threads_started":1}
I20260927 22:56:02.134867 18668 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a3313cf3dc474290fb4eb2f371469a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39545 } }
I20260927 22:56:02.135375 18668 leader_election.cc:302] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2a3313cf3dc474290fb4eb2f371469a; no voters:
I20260927 22:56:02.135900 18668 leader_election.cc:288] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:02.136041 18673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:02.136734 18673 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [term 1 LEADER]: Becoming Leader. State: Replica: d2a3313cf3dc474290fb4eb2f371469a, State: Running, Role: LEADER
I20260927 22:56:02.137283 18673 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a3313cf3dc474290fb4eb2f371469a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39545 } }
I20260927 22:56:02.137991 18668 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:02.140105 18674 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2a3313cf3dc474290fb4eb2f371469a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a3313cf3dc474290fb4eb2f371469a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39545 } } }
I20260927 22:56:02.140316 18675 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a3313cf3dc474290fb4eb2f371469a. Latest consensus state: current_term: 1 leader_uuid: "d2a3313cf3dc474290fb4eb2f371469a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a3313cf3dc474290fb4eb2f371469a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39545 } } }
I20260927 22:56:02.140630 18674 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:02.140676 18675 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:02.146497 18681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:02.151137 18681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:02.158990 18681 catalog_manager.cc:1347] Generated new cluster ID: 75707bc2a0314bd1b1c5ed494d589ebf
I20260927 22:56:02.159090 18681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:02.184193 18681 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:02.185951 18681 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:02.208333 18681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a: Generated new TSK 0
I20260927 22:56:02.209179 18681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:56:02.263110 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39545
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:02.580310 18694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:02.580569 18694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:02.580662 18694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:02.587685 18694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:02.587783 18694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:02.587893 18694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:56:02.597196 18694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:02.598387 18694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:02.599612 18694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:02.611821 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:02.611850 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:02.611917 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:02.612679 18694 server_base.cc:1034] running on GCE node
I20260927 22:56:02.613250 18694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:02.614176 18694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:02.615430 18694 hybrid_clock.cc:648] HybridClock initialized: now 1790549762615385 us; error 50 us; skew 500 ppm
I20260927 22:56:02.615744 18694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:02 dist-test-slave-df53 krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:02.619020 18694 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:56:02.620376 18694 webserver.cc:460] Webserver started at http://127.0.97.1:44151/ using document root <none> and password file <none>
I20260927 22:56:02.621021 18694 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:02.621162 18694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:02.621469 18694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:02.624389 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4"
format_stamp: "Formatted at 2026-09-27 22:56:02 on dist-test-slave-df53"
server_key: "d4942adc06d8bacde230193cffb880fa"
server_key_iv: "b9227daa332567af39a334fdb8b19aa8"
server_key_version: "encryptionkey@0"
I20260927 22:56:02.625141 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4"
format_stamp: "Formatted at 2026-09-27 22:56:02 on dist-test-slave-df53"
server_key: "d4942adc06d8bacde230193cffb880fa"
server_key_iv: "b9227daa332567af39a334fdb8b19aa8"
server_key_version: "encryptionkey@0"
I20260927 22:56:02.629992 18694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260927 22:56:02.633882 18709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:02.635063 18694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260927 22:56:02.635232 18694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4"
format_stamp: "Formatted at 2026-09-27 22:56:02 on dist-test-slave-df53"
server_key: "d4942adc06d8bacde230193cffb880fa"
server_key_iv: "b9227daa332567af39a334fdb8b19aa8"
server_key_version: "encryptionkey@0"
I20260927 22:56:02.635421 18694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:02.662282 18694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:02.666478 18694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:02.666774 18694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:02.667667 18694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:02.668911 18694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:02.669013 18694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:02.669157 18694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:02.669230 18694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:02.701792 18694 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41297
I20260927 22:56:02.701875 18821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41297 every 8 connection(s)
I20260927 22:56:02.703022 18694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:56:02.703853 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 18694
I20260927 22:56:02.704084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:56:02.704654 388 external_mini_cluster.cc:1428] Setting key febe00f62cf290e7c81a3316d592aad0
Sep 27 22:56:02 dist-test-slave-df53 krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:02.723821 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:02.705764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43007 (local address 127.0.97.62:39545)
0927 22:56:02.706237 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:02.706241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:02.707332 (+ 1091us) server_negotiation.cc:407] Connection header received
0927 22:56:02.709449 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:02.709455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:02.709534 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:02.709686 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:02.711282 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:02.712175 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:02.713773 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:02.714172 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:02.714281 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:02.718342 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:02.718347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:02.718350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:02.718382 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:02.720815 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:02.721513 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:02.721521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:02.721529 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:02.721711 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:02.722144 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:02.722150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:02.722153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:02.722436 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:02.722593 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:02.723254 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 22:56:02.723469 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":246,"threads_started":1}
I20260927 22:56:02.724759 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:02.706450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39545 (local address 127.0.97.1:43007)
0927 22:56:02.707188 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:02.707223 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:02.709195 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:02.709930 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:02.709938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:02.710570 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:02.711119 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:02.711136 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:02.712335 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:02.712339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:02.713605 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:02.713643 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:02.714409 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:02.714416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:02.714671 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:02.715316 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:02.715348 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:02.718125 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:02.720956 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:02.720964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:02.720978 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:02.721382 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:02.721822 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:02.721828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:02.721831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:02.722024 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:02.722565 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:02.722572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:02.722877 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 22:56:02.723139 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 22:56:02.723636 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":324,"threads_started":1}
I20260927 22:56:02.726279 18822 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39545
I20260927 22:56:02.726608 18822 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:02.727432 18822 heartbeater.cc:511] Master 127.0.97.62:39545 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 22:56:02.729636 18633 ts_manager.cc:194] Registered new tserver with Master: ae9e7cf6e6d54a39a7a3767d3b8ee7a4 (127.0.97.1:41297)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 22:56:02.731184 18633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:56:02.757130 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39545
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:03.071205 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:03.071483 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:03.071574 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:03.078557 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:03.078655 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:03.078761 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:56:03.087963 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:03.089085 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:03.090291 18828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:03.102790 18833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:03.102825 18834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:03.103216 18836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:03.103926 18828 server_base.cc:1034] running on GCE node
I20260927 22:56:03.104542 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:03.105358 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:03.106581 18828 hybrid_clock.cc:648] HybridClock initialized: now 1790549763106519 us; error 56 us; skew 500 ppm
I20260927 22:56:03.106877 18828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:03 dist-test-slave-df53 krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:03.113240 18828 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:56:03.114797 18828 webserver.cc:460] Webserver started at http://127.0.97.2:35663/ using document root <none> and password file <none>
I20260927 22:56:03.115562 18828 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:03.115684 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:03.115968 18828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:03.119359 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "dd2b4621d013435bae8bbd407b3084bc"
format_stamp: "Formatted at 2026-09-27 22:56:03 on dist-test-slave-df53"
server_key: "cd08f32cd72e5944e20af91a343278bf"
server_key_iv: "a3e8ac8208234c39264bb642a4f532ad"
server_key_version: "encryptionkey@0"
I20260927 22:56:03.120051 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dd2b4621d013435bae8bbd407b3084bc"
format_stamp: "Formatted at 2026-09-27 22:56:03 on dist-test-slave-df53"
server_key: "cd08f32cd72e5944e20af91a343278bf"
server_key_iv: "a3e8ac8208234c39264bb642a4f532ad"
server_key_version: "encryptionkey@0"
I20260927 22:56:03.124651 18828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 22:56:03.127846 18843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:03.128996 18828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 22:56:03.129159 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "dd2b4621d013435bae8bbd407b3084bc"
format_stamp: "Formatted at 2026-09-27 22:56:03 on dist-test-slave-df53"
server_key: "cd08f32cd72e5944e20af91a343278bf"
server_key_iv: "a3e8ac8208234c39264bb642a4f532ad"
server_key_version: "encryptionkey@0"
I20260927 22:56:03.129333 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:03.153049 18828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:03.156996 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:03.157286 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:03.158183 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:03.159379 18828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:03.159459 18828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:03.159580 18828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:03.159631 18828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:03.191977 18828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44859
I20260927 22:56:03.192062 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44859 every 8 connection(s)
I20260927 22:56:03.193219 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:56:03.198385 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 18828
I20260927 22:56:03.198573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:56:03.199084 388 external_mini_cluster.cc:1428] Setting key e722d906fd04736ec820d3301e185295
Sep 27 22:56:03 dist-test-slave-df53 krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:03.214812 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.195635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43195 (local address 127.0.97.62:39545)
0927 22:56:03.195801 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:03.195805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:03.196955 (+ 1150us) server_negotiation.cc:407] Connection header received
0927 22:56:03.199345 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:03.199349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:03.199419 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:03.199505 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:03.201295 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.202090 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.204540 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.204945 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.205043 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.209155 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:03.209160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:03.209164 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:03.209195 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:03.211802 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:03.212491 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:03.212499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:03.212502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:03.212703 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:03.213158 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:03.213164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:03.213167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:03.213432 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:03.213591 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:03.214235 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 22:56:03.214476 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:56:03.215684 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.196035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39545 (local address 127.0.97.2:43195)
0927 22:56:03.196766 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:03.196815 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:03.199019 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:03.199799 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:03.199811 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:03.200479 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:03.201107 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.201126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.202249 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.202257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.204401 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.204418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.205158 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.205166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.205469 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.206182 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:03.206212 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:03.208948 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:03.211949 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:03.211957 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:03.211971 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:03.212371 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:03.212857 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:03.212862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:03.212866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:03.213044 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:03.213567 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:03.213575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:03.213895 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 22:56:03.214157 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 22:56:03.214631 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":324,"threads_started":1}
I20260927 22:56:03.217088 18956 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39545
I20260927 22:56:03.217393 18956 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:03.218240 18956 heartbeater.cc:511] Master 127.0.97.62:39545 requested a full tablet report, sending...
I20260927 22:56:03.219832 18633 ts_manager.cc:194] Registered new tserver with Master: dd2b4621d013435bae8bbd407b3084bc (127.0.97.2:44859)
I20260927 22:56:03.221130 18633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43195
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:56:03.253218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39545
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:03.568465 18961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:03.568735 18961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:03.568827 18961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:03.579728 18961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:03.579870 18961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:03.580058 18961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:56:03.595235 18961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:03.596972 18961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:03.598539 18961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:03.615842 18966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:03.617839 18969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:03.617900 18967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:03.619915 18961 server_base.cc:1034] running on GCE node
I20260927 22:56:03.620654 18961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:03.622044 18961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:03.623315 18961 hybrid_clock.cc:648] HybridClock initialized: now 1790549763623236 us; error 71 us; skew 500 ppm
I20260927 22:56:03.623742 18961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:03 dist-test-slave-df53 krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:03.627493 18961 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:56:03.629173 18961 webserver.cc:460] Webserver started at http://127.0.97.3:45675/ using document root <none> and password file <none>
I20260927 22:56:03.630115 18961 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:03.630254 18961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:03.630553 18961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:03.634792 18961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce"
format_stamp: "Formatted at 2026-09-27 22:56:03 on dist-test-slave-df53"
server_key: "0a37ba15fc4392e681d96b9c67ce096b"
server_key_iv: "41b1b2d57194c1767e57cfa3c9313b3e"
server_key_version: "encryptionkey@0"
I20260927 22:56:03.635978 18961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce"
format_stamp: "Formatted at 2026-09-27 22:56:03 on dist-test-slave-df53"
server_key: "0a37ba15fc4392e681d96b9c67ce096b"
server_key_iv: "41b1b2d57194c1767e57cfa3c9313b3e"
server_key_version: "encryptionkey@0"
I20260927 22:56:03.642622 18961 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 22:56:03.647157 18976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:03.648479 18961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:56:03.648643 18961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce"
format_stamp: "Formatted at 2026-09-27 22:56:03 on dist-test-slave-df53"
server_key: "0a37ba15fc4392e681d96b9c67ce096b"
server_key_iv: "41b1b2d57194c1767e57cfa3c9313b3e"
server_key_version: "encryptionkey@0"
I20260927 22:56:03.648871 18961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:03.664950 18961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:03.668910 18961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:03.669188 18961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:03.670214 18961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:03.671484 18961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:03.671561 18961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:03.671680 18961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:03.671746 18961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:03.704952 18961 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36601
I20260927 22:56:03.705005 19088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36601 every 8 connection(s)
I20260927 22:56:03.706220 18961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:56:03.715260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 18961
I20260927 22:56:03.715452 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:56:03.715999 388 external_mini_cluster.cc:1428] Setting key 201d903fd669b8ccabf341b64de42341
Sep 27 22:56:03 dist-test-slave-df53 krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:03.726168 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.708645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38603 (local address 127.0.97.62:39545)
0927 22:56:03.708815 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:03.708819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:03.709933 (+ 1114us) server_negotiation.cc:407] Connection header received
0927 22:56:03.711754 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:03.711760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:03.711851 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:03.711969 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:03.713397 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.714339 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.716770 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.717249 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.717372 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.721105 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:03.721109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:03.721112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:03.721145 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:03.723634 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:03.724279 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:03.724282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:03.724284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:03.724428 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:03.724781 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:03.724784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:03.724786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:03.725025 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:03.725130 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:03.725768 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 22:56:03.725934 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:56:03.727082 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.709034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39545 (local address 127.0.97.3:38603)
0927 22:56:03.709772 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:03.709807 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:03.711487 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:03.712193 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:03.712202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:03.712691 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:03.713222 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.713238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.714486 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.714492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.716318 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.716332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.717523 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.717533 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.717819 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.718560 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:03.718582 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:03.720930 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:03.723754 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:03.723760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:03.723771 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:03.724185 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:03.724522 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:03.724525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:03.724527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:03.724691 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:03.725127 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:03.725132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:03.725403 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 22:56:03.725689 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 22:56:03.726047 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":279,"threads_started":1}
I20260927 22:56:03.728360 19089 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39545
I20260927 22:56:03.728667 19089 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:03.729331 19089 heartbeater.cc:511] Master 127.0.97.62:39545 requested a full tablet report, sending...
I20260927 22:56:03.730783 18633 ts_manager.cc:194] Registered new tserver with Master: 5dcb0c1cd0ca498784a5d64d4180a3ce (127.0.97.3:36601)
I20260927 22:56:03.731328 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:56:03.731739 18633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38603
I20260927 22:56:03.733906 18822 heartbeater.cc:503] Master 127.0.97.62:39545 was elected leader, sending a full tablet report...
I20260927 22:56:03.749096 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.736024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.0.97.62:39545)
0927 22:56:03.736210 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:03.736218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:03.736665 (+ 447us) server_negotiation.cc:407] Connection header received
0927 22:56:03.736772 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:03.736775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:03.736827 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:03.736904 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:03.738389 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.739240 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.740627 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.741019 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.741114 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.743578 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:03.743585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:03.743596 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:03.743625 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:03.746379 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:03.747102 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:03.747107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:03.747111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:03.747230 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:03.747679 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:03.747684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:03.747687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:03.747978 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:03.748247 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:03.748631 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 22:56:03.748815 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:56:03.752911 18633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:56:03.759665 18633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:56:03.788682 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.771216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44859 (local address 127.0.0.1:46414)
0927 22:56:03.777625 (+ 6409us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:03.777673 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:03.777861 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:03.779264 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:03.779275 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:03.779314 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:03.779821 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.779837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.782087 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.782096 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.784916 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.784935 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.787492 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.787501 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.787777 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.787843 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 22:56:03.787990 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 22:56:03.788174 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6076,"thread_start_us":5951,"threads_started":1}
I20260927 22:56:03.789695 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.771443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.0.97.2:44859)
0927 22:56:03.778305 (+ 6862us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:03.778317 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:03.778352 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:56:03.778460 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:03.778470 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:03.778757 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:03.778980 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:03.780059 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.781909 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.785129 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.786929 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.787077 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.787334 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:03.789267 (+ 1933us) server_negotiation.cc:299] Negotiation successful
0927 22:56:03.789424 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6665,"thread_start_us":6576,"threads_started":1}
I20260927 22:56:03.793711 18891 tablet_service.cc:1467] Processing CreateTablet for tablet 486af2e6e02847b9845f57e23668709a (DEFAULT_TABLE table=test-table [id=9a64b683a42148dd861248f9c46e8baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:03.795187 18891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486af2e6e02847b9845f57e23668709a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:03.797222 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.771684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41297 (local address 127.0.0.1:51184)
0927 22:56:03.788985 (+ 17301us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:03.789009 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:03.789180 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:03.790110 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:03.790114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:03.790138 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:03.791502 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.791513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.792851 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.792860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.794833 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.794845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.796445 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.796452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.796609 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.796640 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:56:03.796718 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 22:56:03.796842 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17126,"mutex_wait_us":135,"thread_start_us":17750,"threads_started":1}
I20260927 22:56:03.797508 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.772393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.0.97.1:41297)
0927 22:56:03.787622 (+ 15229us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:03.787634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:03.789158 (+ 1524us) server_negotiation.cc:407] Connection header received
0927 22:56:03.789349 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:03.789356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:03.789635 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:03.789828 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:03.791704 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.792709 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.794997 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.796226 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.796356 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.796910 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:03.797002 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 22:56:03.797170 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14991,"thread_start_us":15230,"threads_started":1}
I20260927 22:56:03.798310 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.771936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36601 (local address 127.0.0.1:43732)
0927 22:56:03.789716 (+ 17780us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:03.789738 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:03.789850 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:03.790794 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:03.790798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:03.790826 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:03.791276 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.791313 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.793249 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.793253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.795537 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.795551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.797479 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.797487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.797678 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.797708 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:56:03.797788 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 22:56:03.797913 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17619,"mutex_wait_us":1,"thread_start_us":16710,"threads_started":1}
I20260927 22:56:03.798543 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.772651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.0.97.3:36601)
0927 22:56:03.788470 (+ 15819us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:03.788482 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:03.789994 (+ 1512us) server_negotiation.cc:407] Connection header received
0927 22:56:03.790094 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:03.790100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:03.790335 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:03.790519 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:03.791520 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.793055 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.795706 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.797085 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.797203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.797977 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:03.798047 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 22:56:03.798195 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15607,"thread_start_us":15417,"threads_started":1}
I20260927 22:56:03.802064 18757 tablet_service.cc:1467] Processing CreateTablet for tablet 486af2e6e02847b9845f57e23668709a (DEFAULT_TABLE table=test-table [id=9a64b683a42148dd861248f9c46e8baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:03.802064 19024 tablet_service.cc:1467] Processing CreateTablet for tablet 486af2e6e02847b9845f57e23668709a (DEFAULT_TABLE table=test-table [id=9a64b683a42148dd861248f9c46e8baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:03.803460 19024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486af2e6e02847b9845f57e23668709a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:03.803469 18757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486af2e6e02847b9845f57e23668709a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:03.809646 19107 tablet_bootstrap.cc:492] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: Bootstrap starting.
I20260927 22:56:03.812577 19107 tablet_bootstrap.cc:654] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:03.813566 19107 log.cc:824] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:03.817142 19107 tablet_bootstrap.cc:492] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: No bootstrap required, opened a new log
I20260927 22:56:03.817438 19107 ts_tablet_manager.cc:1397] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 22:56:03.818070 19109 tablet_bootstrap.cc:492] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Bootstrap starting.
I20260927 22:56:03.819644 19110 tablet_bootstrap.cc:492] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: Bootstrap starting.
I20260927 22:56:03.821640 19109 tablet_bootstrap.cc:654] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:03.822575 19110 tablet_bootstrap.cc:654] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:03.822901 19109 log.cc:824] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:03.823632 19110 log.cc:824] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:03.825785 19109 tablet_bootstrap.cc:492] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: No bootstrap required, opened a new log
I20260927 22:56:03.826040 19109 ts_tablet_manager.cc:1397] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260927 22:56:03.827541 19110 tablet_bootstrap.cc:492] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: No bootstrap required, opened a new log
I20260927 22:56:03.827790 19110 ts_tablet_manager.cc:1397] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:56:03.827955 19107 raft_consensus.cc:348] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 } } peers { permanent_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41297 } } peers { permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260927 22:56:03.828367 19107 raft_consensus.cc:374] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:03.828480 19107 raft_consensus.cc:729] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2b4621d013435bae8bbd407b3084bc, State: Initialized, Role: FOLLOWER
I20260927 22:56:03.829061 19107 consensus_queue.cc:260] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 } } peers { permanent_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41297 } } peers { permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260927 22:56:03.830679 19107 ts_tablet_manager.cc:1428] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260927 22:56:03.831097 18956 heartbeater.cc:503] Master 127.0.97.62:39545 was elected leader, sending a full tablet report...
I20260927 22:56:03.835680 19109 raft_consensus.cc:348] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 } } peers { permanent_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41297 } } peers { permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260927 22:56:03.836079 19109 raft_consensus.cc:374] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:03.836182 19109 raft_consensus.cc:729] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9e7cf6e6d54a39a7a3767d3b8ee7a4, State: Initialized, Role: FOLLOWER
I20260927 22:56:03.836737 19109 consensus_queue.cc:260] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 } } peers { permanent_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41297 } } peers { permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260927 22:56:03.836999 19110 raft_consensus.cc:348] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 } } peers { permanent_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41297 } } peers { permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260927 22:56:03.837409 19110 raft_consensus.cc:374] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:03.837522 19110 raft_consensus.cc:729] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcb0c1cd0ca498784a5d64d4180a3ce, State: Initialized, Role: FOLLOWER
I20260927 22:56:03.838068 19110 consensus_queue.cc:260] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 } } peers { permanent_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41297 } } peers { permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260927 22:56:03.838340 19109 ts_tablet_manager.cc:1428] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260927 22:56:03.839094 19089 heartbeater.cc:503] Master 127.0.97.62:39545 was elected leader, sending a full tablet report...
I20260927 22:56:03.840601 19110 ts_tablet_manager.cc:1428] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260927 22:56:03.945087 18957 tablet.cc:2369] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:56:03.955343 18823 tablet.cc:2369] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:56:03.957963 19090 tablet.cc:2369] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:04.179965 19114 raft_consensus.cc:482] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:56:04.180287 19114 raft_consensus.cc:504] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 } } peers { permanent_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41297 } } peers { permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260927 22:56:04.182133 19114 leader_election.cc:288] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd2b4621d013435bae8bbd407b3084bc (127.0.97.2:44859), 5dcb0c1cd0ca498784a5d64d4180a3ce (127.0.97.3:36601)
I20260927 22:56:04.188853 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.182571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39969 (local address 127.0.97.2:44859)
0927 22:56:04.183167 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:04.183174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:04.183343 (+ 169us) server_negotiation.cc:407] Connection header received
0927 22:56:04.183560 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:04.183568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:04.183655 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:04.183753 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:04.184423 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.185174 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.187122 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.187887 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.187994 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.188089 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:04.188448 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 22:56:04.188581 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430}
I20260927 22:56:04.188879 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.182748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54585 (local address 127.0.97.3:36601)
0927 22:56:04.182967 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:04.182975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:04.183664 (+ 689us) server_negotiation.cc:407] Connection header received
0927 22:56:04.183828 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:04.183833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:04.183917 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:04.184019 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:04.184601 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.185361 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.187211 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.188002 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.188128 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.188228 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:04.188515 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 22:56:04.188646 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:56:04.188987 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.182361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44859 (local address 127.0.97.1:39969)
0927 22:56:04.183216 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:04.183244 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:04.183411 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:04.183801 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:04.183805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:04.183833 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:04.184274 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:04.184284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.185310 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.185315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:04.186976 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:04.186988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.188108 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.188115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:04.188296 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.188319 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:56:04.188393 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 22:56:04.188537 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":58,"thread_start_us":425,"threads_started":1}
I20260927 22:56:04.189005 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.182610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36601 (local address 127.0.97.1:54585)
0927 22:56:04.183543 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:04.183560 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:04.183690 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:04.184055 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:04.184058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:04.184076 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:04.184422 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:04.184433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.185507 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.185511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:04.187037 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:04.187071 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.188221 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.188225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:04.188331 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.188352 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:56:04.188470 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:56:04.188590 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":55,"thread_start_us":548,"threads_started":1}
I20260927 22:56:04.190148 19044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486af2e6e02847b9845f57e23668709a" candidate_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" is_pre_election: true
I20260927 22:56:04.190155 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486af2e6e02847b9845f57e23668709a" candidate_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd2b4621d013435bae8bbd407b3084bc" is_pre_election: true
I20260927 22:56:04.190690 19044 raft_consensus.cc:2454] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9e7cf6e6d54a39a7a3767d3b8ee7a4 in term 0.
I20260927 22:56:04.190686 18911 raft_consensus.cc:2454] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9e7cf6e6d54a39a7a3767d3b8ee7a4 in term 0.
I20260927 22:56:04.191375 18712 leader_election.cc:302] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9e7cf6e6d54a39a7a3767d3b8ee7a4, dd2b4621d013435bae8bbd407b3084bc; no voters:
I20260927 22:56:04.191857 19114 raft_consensus.cc:2781] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:56:04.192024 19114 raft_consensus.cc:482] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:56:04.192137 19114 raft_consensus.cc:3037] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:04.195353 19114 raft_consensus.cc:504] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 } } peers { permanent_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41297 } } peers { permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260927 22:56:04.196406 19114 leader_election.cc:288] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [CANDIDATE]: Term 1 election: Requested vote from peers dd2b4621d013435bae8bbd407b3084bc (127.0.97.2:44859), 5dcb0c1cd0ca498784a5d64d4180a3ce (127.0.97.3:36601)
I20260927 22:56:04.196522 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486af2e6e02847b9845f57e23668709a" candidate_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd2b4621d013435bae8bbd407b3084bc"
I20260927 22:56:04.196750 18911 raft_consensus.cc:3037] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:04.196733 19044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486af2e6e02847b9845f57e23668709a" candidate_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce"
I20260927 22:56:04.196921 19044 raft_consensus.cc:3037] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:04.199651 18911 raft_consensus.cc:2454] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9e7cf6e6d54a39a7a3767d3b8ee7a4 in term 1.
I20260927 22:56:04.200021 18712 leader_election.cc:302] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9e7cf6e6d54a39a7a3767d3b8ee7a4, dd2b4621d013435bae8bbd407b3084bc; no voters:
I20260927 22:56:04.200245 19044 raft_consensus.cc:2454] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9e7cf6e6d54a39a7a3767d3b8ee7a4 in term 1.
I20260927 22:56:04.200318 19114 raft_consensus.cc:2781] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:04.200822 19114 raft_consensus.cc:686] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 1 LEADER]: Becoming Leader. State: Replica: ae9e7cf6e6d54a39a7a3767d3b8ee7a4, State: Running, Role: LEADER
I20260927 22:56:04.201288 19114 consensus_queue.cc:237] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 } } peers { permanent_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41297 } } peers { permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } }
I20260927 22:56:04.206115 18633 catalog_manager.cc:5495] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9e7cf6e6d54a39a7a3767d3b8ee7a4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9e7cf6e6d54a39a7a3767d3b8ee7a4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 } health_report { overall_health: UNKNOWN } } }
I20260927 22:56:04.255395 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.249830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.0.97.1:41297)
0927 22:56:04.250016 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:04.250022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:04.250149 (+ 127us) server_negotiation.cc:407] Connection header received
0927 22:56:04.250348 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:04.250356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:04.250468 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:04.250643 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:04.251108 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.252084 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.253419 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.253866 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.253959 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.254419 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:04.254884 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:04.255006 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:04.255134 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 22:56:04.255172 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:56:04.263271 19044 raft_consensus.cc:1261] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce [term 1 FOLLOWER]: Refusing update from remote peer ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:04.263357 18911 raft_consensus.cc:1261] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc [term 1 FOLLOWER]: Refusing update from remote peer ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:04.264284 19118 consensus_queue.cc:1035] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcb0c1cd0ca498784a5d64d4180a3ce" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:04.264606 19114 consensus_queue.cc:1035] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd2b4621d013435bae8bbd407b3084bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:04.275727 19122 mvcc.cc:205] Tried to move back new op lower bound from 7334091834412847104 to 7334091834172465152. Current Snapshot: MvccSnapshot[applied={T|T < 7334091834412847104}]
I20260927 22:56:04.288996 18632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:04.289254 18632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:04.292475 18632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2a3313cf3dc474290fb4eb2f371469a: Sending DeleteTablet for 3 replicas of tablet 486af2e6e02847b9845f57e23668709a
I20260927 22:56:04.293550 18891 tablet_service.cc:1514] Processing DeleteTablet for tablet 486af2e6e02847b9845f57e23668709a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46414
I20260927 22:56:04.293562 18757 tablet_service.cc:1514] Processing DeleteTablet for tablet 486af2e6e02847b9845f57e23668709a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51184
I20260927 22:56:04.294556 19024 tablet_service.cc:1514] Processing DeleteTablet for tablet 486af2e6e02847b9845f57e23668709a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43732
I20260927 22:56:04.295297 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 18694
I20260927 22:56:04.312460 19132 tablet_replica.cc:330] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: stopping tablet replica
I20260927 22:56:04.312726 19134 tablet_replica.cc:330] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: stopping tablet replica
I20260927 22:56:04.312893 19132 raft_consensus.cc:2229] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:56:04.313133 19134 raft_consensus.cc:2229] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:04.313555 19132 raft_consensus.cc:2258] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:04.313632 19134 raft_consensus.cc:2258] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:04.314306 19133 tablet_replica.cc:330] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: stopping tablet replica
I20260927 22:56:04.314671 19133 raft_consensus.cc:2229] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:04.315158 19133 raft_consensus.cc:2258] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:04.315927 19134 ts_tablet_manager.cc:1905] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:04.315945 19132 ts_tablet_manager.cc:1905] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:56:04 dist-test-slave-df53 krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:56:04.322392 19133 ts_tablet_manager.cc:1905] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:04.323021 19134 ts_tablet_manager.cc:1918] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:04.323220 19134 log.cc:1196] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/486af2e6e02847b9845f57e23668709a
I20260927 22:56:04.323791 19134 ts_tablet_manager.cc:1939] T 486af2e6e02847b9845f57e23668709a P 5dcb0c1cd0ca498784a5d64d4180a3ce: Deleting consensus metadata
I20260927 22:56:04.323830 19132 ts_tablet_manager.cc:1918] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:04.323974 19132 log.cc:1196] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/486af2e6e02847b9845f57e23668709a
I20260927 22:56:04.324515 19132 ts_tablet_manager.cc:1939] T 486af2e6e02847b9845f57e23668709a P ae9e7cf6e6d54a39a7a3767d3b8ee7a4: Deleting consensus metadata
I20260927 22:56:04.325819 18620 catalog_manager.cc:4841] TS 5dcb0c1cd0ca498784a5d64d4180a3ce (127.0.97.3:36601): tablet 486af2e6e02847b9845f57e23668709a (table test-table [id=9a64b683a42148dd861248f9c46e8baa]) successfully deleted
I20260927 22:56:04.326098 18617 catalog_manager.cc:4841] TS ae9e7cf6e6d54a39a7a3767d3b8ee7a4 (127.0.97.1:41297): tablet 486af2e6e02847b9845f57e23668709a (table test-table [id=9a64b683a42148dd861248f9c46e8baa]) successfully deleted
I20260927 22:56:04.328981 19133 ts_tablet_manager.cc:1918] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:04.329126 19133 log.cc:1196] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/486af2e6e02847b9845f57e23668709a
I20260927 22:56:04.329655 19133 ts_tablet_manager.cc:1939] T 486af2e6e02847b9845f57e23668709a P dd2b4621d013435bae8bbd407b3084bc: Deleting consensus metadata
I20260927 22:56:04.331306 18619 catalog_manager.cc:4841] TS dd2b4621d013435bae8bbd407b3084bc (127.0.97.2:44859): tablet 486af2e6e02847b9845f57e23668709a (table test-table [id=9a64b683a42148dd861248f9c46e8baa]) successfully deleted
I20260927 22:56:04.333432 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.296752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.0.97.1:41297)
0927 22:56:04.296953 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:04.296959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:04.302849 (+ 5890us) server_negotiation.cc:407] Connection header received
0927 22:56:04.303654 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:04.303660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:04.303738 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:04.303810 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:04.307154 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.308333 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.315551 (+ 7218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.316272 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.316373 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.327252 (+ 10879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:04.327262 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:04.327273 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:04.327303 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:04.330535 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.331230 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.331242 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.331248 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.331461 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.331843 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.331848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.331852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.332285 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:04.332591 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:04.332817 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 22:56:04.333052 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:56:04.334107 18817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:04.432828 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 18694
I20260927 22:56:04.451200 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 18828
Sep 27 22:56:04 dist-test-slave-df53 krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:56:04.466637 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.451990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.0.97.2:44859)
0927 22:56:04.452186 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:04.452191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:04.452234 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:56:04.452358 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:04.452369 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:04.452468 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:04.452587 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:04.453961 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.454695 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.456008 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.456419 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.456548 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.461238 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:04.461251 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:04.461266 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:04.461303 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:04.463965 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.464644 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.464650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.464653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.464780 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.465108 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.465111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.465113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.465454 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:04.465761 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:04.466187 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:56:04.466339 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:56:04.467271 18951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:04.554358 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 18828
I20260927 22:56:04.571897 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 18961
Sep 27 22:56:04 dist-test-slave-df53 krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:56:04.586086 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.572637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.0.97.3:36601)
0927 22:56:04.572799 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:04.572804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:04.572822 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:56:04.572940 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:04.572944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:04.573011 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:04.573097 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:04.574421 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.575115 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.576356 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.576752 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.576840 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.581165 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:04.581180 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:04.581197 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:04.581233 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:04.583798 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.584378 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.584385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.584387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.584525 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.584831 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.584834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.584836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.585177 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:04.585375 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:04.585661 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 22:56:04.585813 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:56:04.586735 19084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:04.673627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 18961
I20260927 22:56:04.689951 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 18601
I20260927 22:56:04.691013 18663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:04.798985 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 18601
2026-09-27T22:56:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4436 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T22:56:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:04Z Disabled control of system clock
I20260927 22:56:04.871606 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34887
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:56:05.187608 19143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:05.187912 19143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:05.188000 19143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:05.194960 19143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:05.195055 19143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:05.195093 19143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:05.195143 19143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:05.195194 19143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:05.203893 19143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:05.205014 19143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:05.206671 19143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:05.218367 19148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:05.218420 19149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:05.218380 19151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:05.219437 19143 server_base.cc:1034] running on GCE node
I20260927 22:56:05.220147 19143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:05.221418 19143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:05.222651 19143 hybrid_clock.cc:648] HybridClock initialized: now 1790549765222585 us; error 60 us; skew 500 ppm
I20260927 22:56:05.223011 19143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:56:05.225122 19143 webserver.cc:460] Webserver started at http://127.0.97.62:41463/ using document root <none> and password file <none>
I20260927 22:56:05.225848 19143 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:05.225973 19143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:05.226269 19143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:05.229035 19143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "e0d620af7ba24bd58f7273248aaa185b"
format_stamp: "Formatted at 2026-09-27 22:56:05 on dist-test-slave-df53"
server_key: "57820a1a1c26b71d2aee78a8a6d51f80"
server_key_iv: "8c26d5914acf86356949be80bffbb8a3"
server_key_version: "encryptionkey@0"
I20260927 22:56:05.229875 19143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "e0d620af7ba24bd58f7273248aaa185b"
format_stamp: "Formatted at 2026-09-27 22:56:05 on dist-test-slave-df53"
server_key: "57820a1a1c26b71d2aee78a8a6d51f80"
server_key_iv: "8c26d5914acf86356949be80bffbb8a3"
server_key_version: "encryptionkey@0"
I20260927 22:56:05.234637 19143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:56:05.237979 19157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:05.239252 19143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:56:05.239449 19143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "e0d620af7ba24bd58f7273248aaa185b"
format_stamp: "Formatted at 2026-09-27 22:56:05 on dist-test-slave-df53"
server_key: "57820a1a1c26b71d2aee78a8a6d51f80"
server_key_iv: "8c26d5914acf86356949be80bffbb8a3"
server_key_version: "encryptionkey@0"
I20260927 22:56:05.239655 19143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:05.253358 19143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:05.254163 19143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:05.254446 19143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:05.272372 19143 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34887
I20260927 22:56:05.272452 19208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34887 every 8 connection(s)
I20260927 22:56:05.273702 19143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:05.277129 19209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:05.280715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 19143
I20260927 22:56:05.280889 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:05.281454 388 external_mini_cluster.cc:1428] Setting key 7da82030360c9d3700c452828cff35aa
I20260927 22:56:05.289515 19209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b: Bootstrap starting.
I20260927 22:56:05.292970 19209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:05.294101 19209 log.cc:824] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:05.294517 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:05.283240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.0.97.62:34887)
0927 22:56:05.284153 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:05.284162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:05.284200 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:56:05.285797 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:05.285819 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:05.286244 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:05.286560 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:05.287122 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:05.288359 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:05.290255 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:05.290892 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:05.291031 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:05.292226 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:05.292232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:05.292243 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:05.292732 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:05.292818 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:05.292841 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:05.292922 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:05.293225 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 22:56:05.293541 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":372,"threads_started":1}
I20260927 22:56:05.297065 19209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b: No bootstrap required, opened a new log
I20260927 22:56:05.304003 19209 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d620af7ba24bd58f7273248aaa185b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } }
I20260927 22:56:05.304335 19209 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:05.304441 19209 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d620af7ba24bd58f7273248aaa185b, State: Initialized, Role: FOLLOWER
I20260927 22:56:05.304935 19209 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d620af7ba24bd58f7273248aaa185b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } }
I20260927 22:56:05.305171 19209 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:05.305308 19209 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:05.305477 19209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:05.307698 19209 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d620af7ba24bd58f7273248aaa185b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } }
I20260927 22:56:05.308145 19209 leader_election.cc:302] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d620af7ba24bd58f7273248aaa185b; no voters:
I20260927 22:56:05.308670 19209 leader_election.cc:288] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:05.308806 19214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:05.309489 19214 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [term 1 LEADER]: Becoming Leader. State: Replica: e0d620af7ba24bd58f7273248aaa185b, State: Running, Role: LEADER
I20260927 22:56:05.309998 19214 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d620af7ba24bd58f7273248aaa185b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } }
I20260927 22:56:05.310551 19209 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:05.312573 19216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d620af7ba24bd58f7273248aaa185b. Latest consensus state: current_term: 1 leader_uuid: "e0d620af7ba24bd58f7273248aaa185b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d620af7ba24bd58f7273248aaa185b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } } }
I20260927 22:56:05.313557 19216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:05.312974 19215 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0d620af7ba24bd58f7273248aaa185b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d620af7ba24bd58f7273248aaa185b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } } }
I20260927 22:56:05.315271 19215 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:05.322860 19223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:05.326761 19223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:05.333886 19223 catalog_manager.cc:1347] Generated new cluster ID: ec1f6865a90043b2b33de87f17990a62
I20260927 22:56:05.333992 19223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:05.342792 19223 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:05.343993 19223 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:05.353927 19223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b: Generated new TSK 0
I20260927 22:56:05.354650 19223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:05.366035 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--builtin_ntp_servers=127.0.97.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:56:05.691479 19233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:05.691772 19233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:05.691923 19233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:05.698906 19233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:05.699030 19233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:05.699203 19233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:56:05.707675 19233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:05.708828 19233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:05.710568 19233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:05.723017 19241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:05.723067 19238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:05.723068 19239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:05.723757 19233 server_base.cc:1034] running on GCE node
I20260927 22:56:05.724241 19233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:05.725104 19233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:05.726292 19233 hybrid_clock.cc:648] HybridClock initialized: now 1790549765726255 us; error 29 us; skew 500 ppm
I20260927 22:56:05.726586 19233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:56:05.728623 19233 webserver.cc:460] Webserver started at http://127.0.97.1:39649/ using document root <none> and password file <none>
I20260927 22:56:05.729255 19233 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:05.729364 19233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:05.729780 19233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:05.732451 19233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "f3f70c96978944eda5285b5030ce2b89"
format_stamp: "Formatted at 2026-09-27 22:56:05 on dist-test-slave-df53"
server_key: "6e5432f5878e57fad3a8fb409ffd98ae"
server_key_iv: "08033cb12c3e1d78869120eec1b711c4"
server_key_version: "encryptionkey@0"
I20260927 22:56:05.733196 19233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f3f70c96978944eda5285b5030ce2b89"
format_stamp: "Formatted at 2026-09-27 22:56:05 on dist-test-slave-df53"
server_key: "6e5432f5878e57fad3a8fb409ffd98ae"
server_key_iv: "08033cb12c3e1d78869120eec1b711c4"
server_key_version: "encryptionkey@0"
I20260927 22:56:05.737920 19233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 22:56:05.741086 19247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:05.742273 19233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:05.742449 19233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "f3f70c96978944eda5285b5030ce2b89"
format_stamp: "Formatted at 2026-09-27 22:56:05 on dist-test-slave-df53"
server_key: "6e5432f5878e57fad3a8fb409ffd98ae"
server_key_iv: "08033cb12c3e1d78869120eec1b711c4"
server_key_version: "encryptionkey@0"
I20260927 22:56:05.742695 19233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:05.757174 19233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:05.758028 19233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:05.758297 19233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:05.759171 19233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:05.760532 19233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:05.760660 19233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:05.760779 19233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:05.760865 19233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:05.792402 19233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39583
I20260927 22:56:05.792460 19359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39583 every 8 connection(s)
I20260927 22:56:05.793732 19233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:56:05.802134 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 19233
I20260927 22:56:05.802304 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:56:05.802845 388 external_mini_cluster.cc:1428] Setting key 447e18dfada47dd0f982d16ab5d7b284
I20260927 22:56:05.805729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--builtin_ntp_servers=127.0.97.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:56:05.806627 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:05.796137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34267 (local address 127.0.97.62:34887)
0927 22:56:05.796574 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:05.796579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:05.797699 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 22:56:05.799425 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:05.799431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:05.799520 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:05.799638 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:05.800660 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:05.801385 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:05.803634 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:05.804070 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:05.804209 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:05.805713 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:05.805718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:05.805721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:05.805736 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:05.805786 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:05.805819 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:05.805932 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:05.806193 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 22:56:05.806283 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":242,"threads_started":1}
I20260927 22:56:05.807583 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:05.796633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34887 (local address 127.0.97.1:34267)
0927 22:56:05.797508 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:05.797542 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:05.799170 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:05.799862 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:05.799870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:05.799953 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:05.800499 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:05.800515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:05.801538 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:05.801550 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:05.803480 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:05.803492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:05.804292 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:05.804300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:05.804569 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:05.805307 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:56:05.805311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:05.805361 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:56:05.805469 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:05.805988 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:05.805997 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:05.806003 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:56:05.806182 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 22:56:05.806571 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":279,"threads_started":1}
I20260927 22:56:05.809000 19360 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34887
I20260927 22:56:05.809289 19360 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:05.810206 19360 heartbeater.cc:511] Master 127.0.97.62:34887 requested a full tablet report, sending...
I20260927 22:56:05.812173 19174 ts_manager.cc:194] Registered new tserver with Master: f3f70c96978944eda5285b5030ce2b89 (127.0.97.1:39583)
I20260927 22:56:05.813898 19174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:34267
W20260927 22:56:06.121940 19364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:06.122200 19364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:06.122341 19364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:06.129854 19364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:06.129949 19364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:06.130162 19364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:56:06.139619 19364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:06.140981 19364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:06.142786 19364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:06.155612 19369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:06.155650 19372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:06.155570 19370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:06.156415 19364 server_base.cc:1034] running on GCE node
I20260927 22:56:06.156991 19364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:06.157899 19364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:06.159135 19364 hybrid_clock.cc:648] HybridClock initialized: now 1790549766159072 us; error 53 us; skew 500 ppm
I20260927 22:56:06.159403 19364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:56:06.161664 19364 webserver.cc:460] Webserver started at http://127.0.97.2:36245/ using document root <none> and password file <none>
I20260927 22:56:06.162276 19364 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:06.162358 19364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:06.162616 19364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:06.165247 19364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "9788f994045e412da65ad74d76b0da48"
format_stamp: "Formatted at 2026-09-27 22:56:06 on dist-test-slave-df53"
server_key: "9006b0127a5a839c453ba8f93c8b5c70"
server_key_iv: "a09dcf294fb9e83fb884c5c186d2c016"
server_key_version: "encryptionkey@0"
I20260927 22:56:06.166018 19364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9788f994045e412da65ad74d76b0da48"
format_stamp: "Formatted at 2026-09-27 22:56:06 on dist-test-slave-df53"
server_key: "9006b0127a5a839c453ba8f93c8b5c70"
server_key_iv: "a09dcf294fb9e83fb884c5c186d2c016"
server_key_version: "encryptionkey@0"
I20260927 22:56:06.170698 19364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:56:06.173838 19378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:06.175045 19364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:56:06.175200 19364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "9788f994045e412da65ad74d76b0da48"
format_stamp: "Formatted at 2026-09-27 22:56:06 on dist-test-slave-df53"
server_key: "9006b0127a5a839c453ba8f93c8b5c70"
server_key_iv: "a09dcf294fb9e83fb884c5c186d2c016"
server_key_version: "encryptionkey@0"
I20260927 22:56:06.175411 19364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:06.199738 19364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:06.200517 19364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:06.200800 19364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:06.201869 19364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:06.203284 19364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:06.203369 19364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:06.203508 19364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:06.203595 19364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:06.238309 19364 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40947
I20260927 22:56:06.238361 19490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40947 every 8 connection(s)
I20260927 22:56:06.239557 19364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:56:06.249804 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 19364
I20260927 22:56:06.249991 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:56:06.250401 388 external_mini_cluster.cc:1428] Setting key ba2c9a385070a9b66f1182d316a1765a
I20260927 22:56:06.252146 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.242237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53961 (local address 127.0.97.62:34887)
0927 22:56:06.242415 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.242420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.243939 (+ 1519us) server_negotiation.cc:407] Connection header received
0927 22:56:06.245861 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.245867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.245956 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.246079 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:06.247102 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.247800 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.249334 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.249802 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.249938 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.251390 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:06.251396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:06.251401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:06.251420 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:06.251478 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:06.251495 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:06.251591 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.251825 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.251897 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:56:06.253276 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.242857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34887 (local address 127.0.97.2:53961)
0927 22:56:06.243780 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:06.243829 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:06.245531 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:06.246298 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:06.246306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:06.246398 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:06.246945 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.246961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.247968 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.247973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.249184 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.249195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.250028 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.250035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.250264 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.251007 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:56:06.251013 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:06.251064 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:56:06.251165 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:06.251611 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:06.251621 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:06.251628 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:56:06.251812 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 22:56:06.252160 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":303,"threads_started":1}
I20260927 22:56:06.253371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--builtin_ntp_servers=127.0.97.20:41265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:56:06.254725 19491 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34887
I20260927 22:56:06.255049 19491 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:06.256024 19491 heartbeater.cc:511] Master 127.0.97.62:34887 requested a full tablet report, sending...
I20260927 22:56:06.257444 19174 ts_manager.cc:194] Registered new tserver with Master: 9788f994045e412da65ad74d76b0da48 (127.0.97.2:40947)
I20260927 22:56:06.258279 19174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:53961
W20260927 22:56:06.572180 19494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:06.572487 19494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:06.572654 19494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:06.579779 19494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:06.579896 19494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:06.580078 19494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:56:06.588795 19494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:06.590008 19494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:06.591686 19494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:06.604176 19499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:06.604210 19500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:06.604135 19502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:06.604981 19494 server_base.cc:1034] running on GCE node
I20260927 22:56:06.605512 19494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:06.606479 19494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:06.607703 19494 hybrid_clock.cc:648] HybridClock initialized: now 1790549766607662 us; error 49 us; skew 500 ppm
I20260927 22:56:06.608016 19494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:56:06.610105 19494 webserver.cc:460] Webserver started at http://127.0.97.3:34305/ using document root <none> and password file <none>
I20260927 22:56:06.610785 19494 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:06.610912 19494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:06.611224 19494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:06.613937 19494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "05fb129420894056bcbbc19a851e3ef4"
format_stamp: "Formatted at 2026-09-27 22:56:06 on dist-test-slave-df53"
server_key: "cf22a2290bd75c07af22a58682f83408"
server_key_iv: "d7b0d5845a8d8a1afd54a1da76b5c20f"
server_key_version: "encryptionkey@0"
I20260927 22:56:06.614665 19494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "05fb129420894056bcbbc19a851e3ef4"
format_stamp: "Formatted at 2026-09-27 22:56:06 on dist-test-slave-df53"
server_key: "cf22a2290bd75c07af22a58682f83408"
server_key_iv: "d7b0d5845a8d8a1afd54a1da76b5c20f"
server_key_version: "encryptionkey@0"
I20260927 22:56:06.619611 19494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:56:06.622820 19508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:06.624095 19494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:56:06.624264 19494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "05fb129420894056bcbbc19a851e3ef4"
format_stamp: "Formatted at 2026-09-27 22:56:06 on dist-test-slave-df53"
server_key: "cf22a2290bd75c07af22a58682f83408"
server_key_iv: "d7b0d5845a8d8a1afd54a1da76b5c20f"
server_key_version: "encryptionkey@0"
I20260927 22:56:06.624480 19494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:06.641093 19494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:06.641875 19494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:06.642141 19494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:06.643038 19494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:06.644389 19494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:06.644506 19494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:06.644635 19494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:06.644721 19494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:06.677496 19494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36149
I20260927 22:56:06.677577 19620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36149 every 8 connection(s)
I20260927 22:56:06.678771 19494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:56:06.687045 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 19494
I20260927 22:56:06.687234 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:56:06.687651 388 external_mini_cluster.cc:1428] Setting key e508880321fd762d85088faca8d21e22
I20260927 22:56:06.691396 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.681263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33527 (local address 127.0.97.62:34887)
0927 22:56:06.681446 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.681451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.682894 (+ 1443us) server_negotiation.cc:407] Connection header received
0927 22:56:06.684641 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.684647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.684739 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.684858 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:06.685961 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.686699 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.688526 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.689073 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.689233 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.690731 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:06.690736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:06.690739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:06.690755 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:06.690784 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:06.690795 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:06.690872 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.691077 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.691148 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:56:06.692278 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.681822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34887 (local address 127.0.97.3:33527)
0927 22:56:06.682739 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:06.682773 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:06.684375 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:06.685063 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:06.685070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:06.685182 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:06.685788 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.685805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.686877 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.686889 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.688386 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.688399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.689337 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.689345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.689631 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.690457 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:56:06.690460 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:06.690494 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:56:06.690566 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:06.690888 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:06.690895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:06.690900 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:56:06.691034 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 22:56:06.691288 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":273,"threads_started":1}
I20260927 22:56:06.693531 19621 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34887
I20260927 22:56:06.693854 19621 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:06.694533 19621 heartbeater.cc:511] Master 127.0.97.62:34887 requested a full tablet report, sending...
I20260927 22:56:06.695871 19174 ts_manager.cc:194] Registered new tserver with Master: 05fb129420894056bcbbc19a851e3ef4 (127.0.97.3:36149)
I20260927 22:56:06.696565 19174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:33527
I20260927 22:56:06.702628 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:56:06.712206 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.706569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.0.97.62:34887)
0927 22:56:06.706749 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.706754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.707259 (+ 505us) server_negotiation.cc:407] Connection header received
0927 22:56:06.707382 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.707385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.707462 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.707537 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:06.707968 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.708635 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.710043 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.710397 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.710551 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.711660 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:06.711664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:06.711668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:06.711683 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:06.711710 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:06.711722 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:06.711822 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.711967 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.712005 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:56:06.715859 19174 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:56:06.722692 19174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:56:06.746739 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.733740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36149 (local address 127.0.0.1:33698)
0927 22:56:06.734563 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:06.734616 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:06.734778 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:06.737343 (+ 2565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:06.737351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:06.737389 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:06.737933 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.737948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.740439 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.740448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.743202 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.743221 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.745732 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.745742 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.745983 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.746039 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:56:06.746147 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:56:06.746355 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":30,"thread_start_us":327,"threads_started":1}
I20260927 22:56:06.747321 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.733771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40947 (local address 127.0.0.1:36314)
0927 22:56:06.735333 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:06.735353 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:06.735491 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:06.736574 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:06.736586 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:06.736623 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:06.737111 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.737128 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.739678 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.739686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.743025 (+ 3339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.743045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.746040 (+ 2995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.746048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.746400 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.746464 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 22:56:06.746616 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 22:56:06.746806 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"mutex_wait_us":4,"thread_start_us":1724,"threads_started":1}
I20260927 22:56:06.748709 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.734389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.0.97.2:40947)
0927 22:56:06.735058 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.735067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.735629 (+ 562us) server_negotiation.cc:407] Connection header received
0927 22:56:06.735729 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.735736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.735992 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.736197 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:06.737368 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.739475 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.743322 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.745250 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.745467 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.745832 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.747940 (+ 2108us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.748179 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":372,"threads_started":1}
I20260927 22:56:06.749082 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.733876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.0.97.3:36149)
0927 22:56:06.735553 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.735561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.735592 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:56:06.735689 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.735696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.735933 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.738160 (+ 2227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:06.738474 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.740176 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.743507 (+ 3331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.745442 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.747875 (+ 2433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.748220 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.748298 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.748526 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1494,"thread_start_us":1337,"threads_started":1}
I20260927 22:56:06.752676 19556 tablet_service.cc:1467] Processing CreateTablet for tablet da15125c78304255b215f2440a5e22a2 (DEFAULT_TABLE table=test-table [id=45a864a752674b0a89c91b83fd1e3a46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:06.752746 19426 tablet_service.cc:1467] Processing CreateTablet for tablet da15125c78304255b215f2440a5e22a2 (DEFAULT_TABLE table=test-table [id=45a864a752674b0a89c91b83fd1e3a46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:06.754258 19426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da15125c78304255b215f2440a5e22a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:06.754494 19556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da15125c78304255b215f2440a5e22a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:06.756456 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.735296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39583 (local address 127.0.0.1:57490)
0927 22:56:06.747609 (+ 12313us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:06.747631 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:06.747752 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:06.750092 (+ 2340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:06.750098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:06.750126 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:06.750287 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.750299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.752102 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.752107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.753898 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.753913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.755701 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.755708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.755898 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.755949 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:56:06.756047 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:56:06.756184 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12190,"thread_start_us":14325,"threads_started":1}
I20260927 22:56:06.756631 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.735732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.0.97.1:39583)
0927 22:56:06.749407 (+ 13675us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.749418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.749465 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:56:06.749577 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.749607 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.749853 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.750089 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:06.750429 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.751931 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.754124 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.755526 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.755671 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.755890 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.756107 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.756260 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13531,"thread_start_us":13438,"threads_started":1}
I20260927 22:56:06.759675 19295 tablet_service.cc:1467] Processing CreateTablet for tablet da15125c78304255b215f2440a5e22a2 (DEFAULT_TABLE table=test-table [id=45a864a752674b0a89c91b83fd1e3a46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:06.760891 19295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da15125c78304255b215f2440a5e22a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:06.765933 19639 tablet_bootstrap.cc:492] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: Bootstrap starting.
I20260927 22:56:06.768843 19639 tablet_bootstrap.cc:654] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:06.769861 19639 log.cc:824] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:06.771072 19641 tablet_bootstrap.cc:492] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: Bootstrap starting.
I20260927 22:56:06.773888 19642 tablet_bootstrap.cc:492] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: Bootstrap starting.
I20260927 22:56:06.775635 19641 tablet_bootstrap.cc:654] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:06.776461 19642 tablet_bootstrap.cc:654] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:06.776507 19639 tablet_bootstrap.cc:492] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: No bootstrap required, opened a new log
I20260927 22:56:06.776753 19639 ts_tablet_manager.cc:1397] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 22:56:06.776811 19641 log.cc:824] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:06.777534 19642 log.cc:824] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:06.779793 19641 tablet_bootstrap.cc:492] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: No bootstrap required, opened a new log
I20260927 22:56:06.780031 19641 ts_tablet_manager.cc:1397] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:56:06.780267 19642 tablet_bootstrap.cc:492] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: No bootstrap required, opened a new log
I20260927 22:56:06.780503 19642 ts_tablet_manager.cc:1397] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 22:56:06.787374 19642 raft_consensus.cc:348] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 } } peers { permanent_uuid: "9788f994045e412da65ad74d76b0da48" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40947 } } peers { permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 } }
I20260927 22:56:06.787422 19639 raft_consensus.cc:348] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 } } peers { permanent_uuid: "9788f994045e412da65ad74d76b0da48" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40947 } } peers { permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 } }
I20260927 22:56:06.787729 19642 raft_consensus.cc:374] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:06.787820 19642 raft_consensus.cc:729] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f70c96978944eda5285b5030ce2b89, State: Initialized, Role: FOLLOWER
I20260927 22:56:06.787885 19639 raft_consensus.cc:374] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:06.787999 19639 raft_consensus.cc:729] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fb129420894056bcbbc19a851e3ef4, State: Initialized, Role: FOLLOWER
I20260927 22:56:06.788547 19642 consensus_queue.cc:260] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 } } peers { permanent_uuid: "9788f994045e412da65ad74d76b0da48" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40947 } } peers { permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 } }
I20260927 22:56:06.788575 19639 consensus_queue.cc:260] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 } } peers { permanent_uuid: "9788f994045e412da65ad74d76b0da48" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40947 } } peers { permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 } }
I20260927 22:56:06.789611 19360 heartbeater.cc:503] Master 127.0.97.62:34887 was elected leader, sending a full tablet report...
I20260927 22:56:06.790268 19642 ts_tablet_manager.cc:1428] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260927 22:56:06.790427 19641 raft_consensus.cc:348] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 } } peers { permanent_uuid: "9788f994045e412da65ad74d76b0da48" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40947 } } peers { permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 } }
I20260927 22:56:06.790943 19641 raft_consensus.cc:374] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:06.791213 19621 heartbeater.cc:503] Master 127.0.97.62:34887 was elected leader, sending a full tablet report...
I20260927 22:56:06.792120 19641 raft_consensus.cc:729] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9788f994045e412da65ad74d76b0da48, State: Initialized, Role: FOLLOWER
I20260927 22:56:06.792841 19641 consensus_queue.cc:260] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 } } peers { permanent_uuid: "9788f994045e412da65ad74d76b0da48" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40947 } } peers { permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 } }
I20260927 22:56:06.793737 19639 ts_tablet_manager.cc:1428] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
W20260927 22:56:06.795802 19361 tablet.cc:2369] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:06.796459 19491 heartbeater.cc:503] Master 127.0.97.62:34887 was elected leader, sending a full tablet report...
I20260927 22:56:06.797785 19641 ts_tablet_manager.cc:1428] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260927 22:56:06.809934 19647 raft_consensus.cc:482] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:56:06.810135 19647 raft_consensus.cc:504] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 } } peers { permanent_uuid: "9788f994045e412da65ad74d76b0da48" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40947 } } peers { permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 } }
I20260927 22:56:06.812736 19647 leader_election.cc:288] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05fb129420894056bcbbc19a851e3ef4 (127.0.97.3:36149), f3f70c96978944eda5285b5030ce2b89 (127.0.97.1:39583)
I20260927 22:56:06.821022 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.812167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41283 (local address 127.0.97.1:39583)
0927 22:56:06.812474 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.812479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.814113 (+ 1634us) server_negotiation.cc:407] Connection header received
0927 22:56:06.814262 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.814269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.814355 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.814449 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:06.815097 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.815915 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.818561 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.819838 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.819988 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.820080 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.820642 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.820767 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260927 22:56:06.821025 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.812130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33571 (local address 127.0.97.3:36149)
0927 22:56:06.812329 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.812337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.814314 (+ 1977us) server_negotiation.cc:407] Connection header received
0927 22:56:06.814479 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.814484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.814556 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.814679 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:06.815322 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.816208 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.818655 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.819890 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.820003 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.820085 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.820642 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.820767 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:56:06.821167 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.812116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36149 (local address 127.0.97.2:33571)
0927 22:56:06.814206 (+ 2090us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:06.814222 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:06.814341 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:06.814717 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:06.814720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:06.814739 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:06.815156 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.815168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.816410 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.816418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.818480 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.818494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.820067 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.820072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.820376 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.820509 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 22:56:06.820600 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:56:06.820818 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1930,"mutex_wait_us":201,"thread_start_us":1463,"threads_started":1}
I20260927 22:56:06.821434 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.811989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39583 (local address 127.0.97.2:41283)
0927 22:56:06.813990 (+ 2001us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:06.814013 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:06.814137 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:06.814465 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:06.814470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:06.814495 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:06.814956 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.814966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.816039 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.816044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.818379 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.818394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.820063 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.820069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.820372 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.820509 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 22:56:06.820600 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:56:06.820818 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1819,"mutex_wait_us":163,"thread_start_us":1870,"threads_started":1}
I20260927 22:56:06.822358 19576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da15125c78304255b215f2440a5e22a2" candidate_uuid: "9788f994045e412da65ad74d76b0da48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05fb129420894056bcbbc19a851e3ef4" is_pre_election: true
I20260927 22:56:06.822500 19315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da15125c78304255b215f2440a5e22a2" candidate_uuid: "9788f994045e412da65ad74d76b0da48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f70c96978944eda5285b5030ce2b89" is_pre_election: true
I20260927 22:56:06.822877 19576 raft_consensus.cc:2454] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9788f994045e412da65ad74d76b0da48 in term 0.
I20260927 22:56:06.822996 19315 raft_consensus.cc:2454] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9788f994045e412da65ad74d76b0da48 in term 0.
I20260927 22:56:06.823606 19382 leader_election.cc:302] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fb129420894056bcbbc19a851e3ef4, 9788f994045e412da65ad74d76b0da48; no voters:
I20260927 22:56:06.823957 19647 raft_consensus.cc:2781] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:56:06.824110 19647 raft_consensus.cc:482] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:56:06.824221 19647 raft_consensus.cc:3037] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:06.826445 19647 raft_consensus.cc:504] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 } } peers { permanent_uuid: "9788f994045e412da65ad74d76b0da48" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40947 } } peers { permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 } }
I20260927 22:56:06.826983 19647 leader_election.cc:288] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [CANDIDATE]: Term 1 election: Requested vote from peers 05fb129420894056bcbbc19a851e3ef4 (127.0.97.3:36149), f3f70c96978944eda5285b5030ce2b89 (127.0.97.1:39583)
I20260927 22:56:06.827463 19315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da15125c78304255b215f2440a5e22a2" candidate_uuid: "9788f994045e412da65ad74d76b0da48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f70c96978944eda5285b5030ce2b89"
I20260927 22:56:06.827497 19576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da15125c78304255b215f2440a5e22a2" candidate_uuid: "9788f994045e412da65ad74d76b0da48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05fb129420894056bcbbc19a851e3ef4"
I20260927 22:56:06.827692 19315 raft_consensus.cc:3037] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:06.827690 19576 raft_consensus.cc:3037] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:06.830868 19576 raft_consensus.cc:2454] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9788f994045e412da65ad74d76b0da48 in term 1.
I20260927 22:56:06.830862 19315 raft_consensus.cc:2454] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9788f994045e412da65ad74d76b0da48 in term 1.
I20260927 22:56:06.831235 19380 leader_election.cc:302] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9788f994045e412da65ad74d76b0da48, f3f70c96978944eda5285b5030ce2b89; no voters:
I20260927 22:56:06.831552 19647 raft_consensus.cc:2781] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:06.831967 19647 raft_consensus.cc:686] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 1 LEADER]: Becoming Leader. State: Replica: 9788f994045e412da65ad74d76b0da48, State: Running, Role: LEADER
I20260927 22:56:06.832471 19647 consensus_queue.cc:237] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 } } peers { permanent_uuid: "9788f994045e412da65ad74d76b0da48" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40947 } } peers { permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 } }
I20260927 22:56:06.837055 19173 catalog_manager.cc:5495] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9788f994045e412da65ad74d76b0da48 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9788f994045e412da65ad74d76b0da48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9788f994045e412da65ad74d76b0da48" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 } health_report { overall_health: UNKNOWN } } }
I20260927 22:56:06.869225 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.863914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.0.97.2:40947)
0927 22:56:06.864111 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.864117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.864135 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:56:06.864301 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.864305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.864416 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.864560 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:06.864954 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.865690 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.867048 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.867455 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.867590 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.868181 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:06.868641 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:06.868823 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.868971 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.869013 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:56:06.876832 19576 raft_consensus.cc:1261] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4 [term 1 FOLLOWER]: Refusing update from remote peer 9788f994045e412da65ad74d76b0da48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:06.876832 19315 raft_consensus.cc:1261] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89 [term 1 FOLLOWER]: Refusing update from remote peer 9788f994045e412da65ad74d76b0da48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:06.877858 19650 consensus_queue.cc:1035] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f70c96978944eda5285b5030ce2b89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:06.878167 19647 consensus_queue.cc:1035] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05fb129420894056bcbbc19a851e3ef4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:06.892496 19654 mvcc.cc:205] Tried to move back new op lower bound from 7334091845118517248 to 7334091844950368256. Current Snapshot: MvccSnapshot[applied={T|T < 7334091845118517248}]
I20260927 22:56:06.900279 19173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:06.900565 19173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:06.904081 19173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0d620af7ba24bd58f7273248aaa185b: Sending DeleteTablet for 3 replicas of tablet da15125c78304255b215f2440a5e22a2
I20260927 22:56:06.905181 19426 tablet_service.cc:1514] Processing DeleteTablet for tablet da15125c78304255b215f2440a5e22a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:06 UTC) from {username='slave'} at 127.0.0.1:36314
I20260927 22:56:06.905175 19556 tablet_service.cc:1514] Processing DeleteTablet for tablet da15125c78304255b215f2440a5e22a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:06 UTC) from {username='slave'} at 127.0.0.1:33698
I20260927 22:56:06.905249 19295 tablet_service.cc:1514] Processing DeleteTablet for tablet da15125c78304255b215f2440a5e22a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:06 UTC) from {username='slave'} at 127.0.0.1:57490
I20260927 22:56:06.907630 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 19233
I20260927 22:56:06.916445 19663 tablet_replica.cc:330] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: stopping tablet replica
I20260927 22:56:06.917133 19663 raft_consensus.cc:2229] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:56:06.918102 19663 raft_consensus.cc:2258] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:06.918370 19664 tablet_replica.cc:330] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: stopping tablet replica
I20260927 22:56:06.918795 19664 raft_consensus.cc:2229] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:06.919358 19664 raft_consensus.cc:2258] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:06.920113 19662 tablet_replica.cc:330] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: stopping tablet replica
I20260927 22:56:06.920493 19662 raft_consensus.cc:2229] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:06.920915 19662 raft_consensus.cc:2258] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:06.922354 19663 ts_tablet_manager.cc:1905] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:06.922889 19662 ts_tablet_manager.cc:1905] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:06.923707 19664 ts_tablet_manager.cc:1905] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:06.929904 19662 ts_tablet_manager.cc:1918] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:06.930060 19662 log.cc:1196] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/da15125c78304255b215f2440a5e22a2
I20260927 22:56:06.930728 19662 ts_tablet_manager.cc:1939] T da15125c78304255b215f2440a5e22a2 P 05fb129420894056bcbbc19a851e3ef4: Deleting consensus metadata
I20260927 22:56:06.932662 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.909195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.0.97.1:39583)
0927 22:56:06.909417 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.909426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.923754 (+ 14328us) server_negotiation.cc:407] Connection header received
0927 22:56:06.924254 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.924261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.924374 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.924456 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:06.925438 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.926605 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.928566 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.929232 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.929325 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.930849 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:06.930858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:06.930875 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:06.931629 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:06.931687 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:06.931710 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:06.931821 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.932157 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.932262 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:56:06.933303 19161 catalog_manager.cc:4841] TS 05fb129420894056bcbbc19a851e3ef4 (127.0.97.3:36149): tablet da15125c78304255b215f2440a5e22a2 (table test-table [id=45a864a752674b0a89c91b83fd1e3a46]) successfully deleted
I20260927 22:56:06.934690 19355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:06.936448 19663 ts_tablet_manager.cc:1918] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:06.936617 19663 log.cc:1196] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/da15125c78304255b215f2440a5e22a2
I20260927 22:56:06.937150 19663 ts_tablet_manager.cc:1939] T da15125c78304255b215f2440a5e22a2 P 9788f994045e412da65ad74d76b0da48: Deleting consensus metadata
I20260927 22:56:06.938804 19160 catalog_manager.cc:4841] TS 9788f994045e412da65ad74d76b0da48 (127.0.97.2:40947): tablet da15125c78304255b215f2440a5e22a2 (table test-table [id=45a864a752674b0a89c91b83fd1e3a46]) successfully deleted
I20260927 22:56:07.024344 19664 ts_tablet_manager.cc:1918] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:07.024618 19664 log.cc:1196] T da15125c78304255b215f2440a5e22a2 P f3f70c96978944eda5285b5030ce2b89: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/da15125c78304255b215f2440a5e22a2
I20260927 22:56:07.024969 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 19233
I20260927 22:56:07.042587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 19364
W20260927 22:56:07.042659 19159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39583 (error 108)
I20260927 22:56:07.044610 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.044080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39583 (local address 127.0.0.1:57500)
0927 22:56:07.044251 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:07.044375 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 22:56:07.044950 19159 catalog_manager.cc:4584] TS f3f70c96978944eda5285b5030ce2b89 (127.0.97.1:39583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da15125c78304255b215f2440a5e22a2: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39583: connect: Connection refused (error 111)
I20260927 22:56:07.050794 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.043538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36334 (local address 127.0.97.2:40947)
0927 22:56:07.044190 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:07.044197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:07.044216 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:56:07.044280 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:07.044284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:07.044394 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:07.044536 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:07.045008 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.045910 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.047735 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.048126 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.048212 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:07.049336 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:07.049345 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:07.049357 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:07.050057 (+ 700us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:07.050107 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:07.050127 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:07.050213 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:07.050372 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 22:56:07.050454 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524}
I20260927 22:56:07.051483 19486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:07.098623 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.098158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39583 (local address 127.0.0.1:57516)
0927 22:56:07.098374 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:07.098473 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 22:56:07.099697 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.099314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39583 (local address 127.0.0.1:57522)
0927 22:56:07.099481 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:07.099553 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:56:07.160925 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.160449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39583 (local address 127.0.0.1:57528)
0927 22:56:07.160643 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:07.160740 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 22:56:07.161437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 19364
I20260927 22:56:07.162632 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.162214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39583 (local address 127.0.0.1:57544)
0927 22:56:07.162357 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:07.162444 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 22:56:07.179100 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 19494
I20260927 22:56:07.186128 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.179772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.97.3:36149)
0927 22:56:07.179966 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:07.179973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:07.180076 (+ 103us) server_negotiation.cc:407] Connection header received
0927 22:56:07.180297 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:07.180303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:07.180443 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:07.180554 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:07.180944 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.181752 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.183105 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.183512 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.183654 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:07.184703 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:07.184712 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:07.184726 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:07.185308 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:07.185363 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:07.185382 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:07.185504 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:07.185756 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 22:56:07.185872 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:56:07.186779 19616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:07.244081 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.243538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39583 (local address 127.0.0.1:57558)
0927 22:56:07.243830 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:07.243939 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260927 22:56:07.245076 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.244676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39583 (local address 127.0.0.1:57564)
0927 22:56:07.244874 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:07.244939 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 22:56:07.274365 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 19494
I20260927 22:56:07.290541 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 19143
I20260927 22:56:07.291544 19204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:07.399562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 19143
2026-09-27T22:56:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2599 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:07 dist-test-slave-df53 krb5kdc[19673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:07 dist-test-slave-df53 krb5kdc[19673](info): set up 2 sockets
Sep 27 22:56:07 dist-test-slave-df53 krb5kdc[19673](info): commencing operation
krb5kdc: starting...
W20260927 22:56:08.497769 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:08 dist-test-slave-df53 krb5kdc[19673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:56:08.726612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44321
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:09.040247 19686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:09.040513 19686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:09.040688 19686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:09.047683 19686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:09.047787 19686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:09.047919 19686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:09.048024 19686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:09.048106 19686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:09.057374 19686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:09.058597 19686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:09.059809 19686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:09.071333 19694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:09.071400 19691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:09.071905 19692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:09.071928 19686 server_base.cc:1034] running on GCE node
I20260927 22:56:09.072741 19686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:09.074064 19686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:09.075294 19686 hybrid_clock.cc:648] HybridClock initialized: now 1790549769075249 us; error 33 us; skew 500 ppm
I20260927 22:56:09.075626 19686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:09 dist-test-slave-df53 krb5kdc[19673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:09.083180 19686 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:56:09.084610 19686 webserver.cc:460] Webserver started at http://127.0.97.62:40393/ using document root <none> and password file <none>
I20260927 22:56:09.085281 19686 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:09.085421 19686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:09.085803 19686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:09.088562 19686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "2b493db9a9d64ee0896c219ae5ba8ddd"
format_stamp: "Formatted at 2026-09-27 22:56:09 on dist-test-slave-df53"
server_key: "bceb2a663254725c5ca1ad94639a34bc"
server_key_iv: "b2df4fd269b9e98d4878c796dbba39fa"
server_key_version: "encryptionkey@0"
I20260927 22:56:09.089270 19686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "2b493db9a9d64ee0896c219ae5ba8ddd"
format_stamp: "Formatted at 2026-09-27 22:56:09 on dist-test-slave-df53"
server_key: "bceb2a663254725c5ca1ad94639a34bc"
server_key_iv: "b2df4fd269b9e98d4878c796dbba39fa"
server_key_version: "encryptionkey@0"
I20260927 22:56:09.094113 19686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260927 22:56:09.097304 19701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:09.098565 19686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:09.098742 19686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "2b493db9a9d64ee0896c219ae5ba8ddd"
format_stamp: "Formatted at 2026-09-27 22:56:09 on dist-test-slave-df53"
server_key: "bceb2a663254725c5ca1ad94639a34bc"
server_key_iv: "b2df4fd269b9e98d4878c796dbba39fa"
server_key_version: "encryptionkey@0"
I20260927 22:56:09.098924 19686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:09.120071 19686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:09.124253 19686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:09.124534 19686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:09.142589 19686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44321
I20260927 22:56:09.142625 19752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44321 every 8 connection(s)
I20260927 22:56:09.143903 19686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:09.147356 19753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:09.148254 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 19686
I20260927 22:56:09.148433 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:09.148964 388 external_mini_cluster.cc:1428] Setting key 96c1004c187e5876768b87be49b01e96
I20260927 22:56:09.161161 19753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd: Bootstrap starting.
Sep 27 22:56:09 dist-test-slave-df53 krb5kdc[19673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:09.164589 19753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:09.165656 19753 log.cc:824] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:09.168203 19753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd: No bootstrap required, opened a new log
I20260927 22:56:09.171552 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:09.150822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.0.97.62:44321)
0927 22:56:09.151595 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:09.151606 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:09.151644 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:56:09.153518 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:09.153540 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:09.153967 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:09.154292 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:09.155955 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:09.157180 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:09.159063 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:09.159684 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:09.159814 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:09.164390 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:09.164398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:09.164411 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:09.164454 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:09.167590 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:09.168287 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:09.168293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:09.168320 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:09.168498 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:09.168869 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:09.168875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:09.168879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:09.169356 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:09.169720 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:09.170142 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 22:56:09.170592 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":358,"threads_started":1}
I20260927 22:56:09.175942 19753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b493db9a9d64ee0896c219ae5ba8ddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44321 } }
I20260927 22:56:09.176314 19753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:09.176414 19753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b493db9a9d64ee0896c219ae5ba8ddd, State: Initialized, Role: FOLLOWER
I20260927 22:56:09.177004 19753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b493db9a9d64ee0896c219ae5ba8ddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44321 } }
I20260927 22:56:09.177268 19753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:09.177408 19753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:09.177559 19753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:09.179522 19753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b493db9a9d64ee0896c219ae5ba8ddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44321 } }
I20260927 22:56:09.179910 19753 leader_election.cc:302] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b493db9a9d64ee0896c219ae5ba8ddd; no voters:
I20260927 22:56:09.180416 19753 leader_election.cc:288] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:09.180510 19758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:09.181265 19758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [term 1 LEADER]: Becoming Leader. State: Replica: 2b493db9a9d64ee0896c219ae5ba8ddd, State: Running, Role: LEADER
I20260927 22:56:09.181825 19758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b493db9a9d64ee0896c219ae5ba8ddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44321 } }
I20260927 22:56:09.182456 19753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:09.184697 19760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b493db9a9d64ee0896c219ae5ba8ddd. Latest consensus state: current_term: 1 leader_uuid: "2b493db9a9d64ee0896c219ae5ba8ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b493db9a9d64ee0896c219ae5ba8ddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44321 } } }
I20260927 22:56:09.185101 19760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:09.184885 19759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b493db9a9d64ee0896c219ae5ba8ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b493db9a9d64ee0896c219ae5ba8ddd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44321 } } }
I20260927 22:56:09.185276 19759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:09.189924 19763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:09.195190 19763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:09.203532 19763 catalog_manager.cc:1347] Generated new cluster ID: 29bbdfac52974d189ba4787ef904beda
I20260927 22:56:09.203645 19763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:09.214876 19763 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:09.216279 19763 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:09.225504 19763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd: Generated new TSK 0
I20260927 22:56:09.226398 19763 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 22:56:09.277472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44321
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:09.600508 19779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:09.600760 19779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:09.600857 19779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:09.608304 19779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:09.608422 19779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:09.608533 19779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:56:09.617978 19779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:09.619163 19779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:09.620349 19779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:09.632997 19785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:09.632934 19787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:09.633401 19784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:09.633931 19779 server_base.cc:1034] running on GCE node
I20260927 22:56:09.634429 19779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:09.635231 19779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:09.636456 19779 hybrid_clock.cc:648] HybridClock initialized: now 1790549769636405 us; error 61 us; skew 500 ppm
I20260927 22:56:09.636744 19779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:09 dist-test-slave-df53 krb5kdc[19673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:09.639840 19779 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 22:56:09.641126 19779 webserver.cc:460] Webserver started at http://127.0.97.1:37959/ using document root <none> and password file <none>
I20260927 22:56:09.641788 19779 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:09.641883 19779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:09.642124 19779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:09.644749 19779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "2ee789601fc642a8befc69a64f3b8f7d"
format_stamp: "Formatted at 2026-09-27 22:56:09 on dist-test-slave-df53"
server_key: "5cc212e48fdc8a446b3ca3ca65ef7b78"
server_key_iv: "30829428c05df75cd32f4d244e022bf4"
server_key_version: "encryptionkey@0"
I20260927 22:56:09.645452 19779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2ee789601fc642a8befc69a64f3b8f7d"
format_stamp: "Formatted at 2026-09-27 22:56:09 on dist-test-slave-df53"
server_key: "5cc212e48fdc8a446b3ca3ca65ef7b78"
server_key_iv: "30829428c05df75cd32f4d244e022bf4"
server_key_version: "encryptionkey@0"
I20260927 22:56:09.650170 19779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:56:09.653288 19794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:09.654525 19779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:09.654650 19779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "2ee789601fc642a8befc69a64f3b8f7d"
format_stamp: "Formatted at 2026-09-27 22:56:09 on dist-test-slave-df53"
server_key: "5cc212e48fdc8a446b3ca3ca65ef7b78"
server_key_iv: "30829428c05df75cd32f4d244e022bf4"
server_key_version: "encryptionkey@0"
I20260927 22:56:09.654811 19779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:09.674242 19779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:09.678006 19779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:09.678243 19779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:09.679098 19779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:09.680361 19779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:09.680442 19779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:09.680538 19779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:09.680583 19779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:09.713371 19779 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32929
I20260927 22:56:09.713471 19906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32929 every 8 connection(s)
I20260927 22:56:09.714680 19779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:56:09.720503 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 19779
I20260927 22:56:09.720679 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:56:09.721125 388 external_mini_cluster.cc:1428] Setting key 76e838cea5f6a06e411689e04fc55152
Sep 27 22:56:09 dist-test-slave-df53 krb5kdc[19673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:09.736464 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:09.717038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54087 (local address 127.0.97.62:44321)
0927 22:56:09.717516 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:09.717521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:09.718297 (+ 776us) server_negotiation.cc:407] Connection header received
0927 22:56:09.720112 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:09.720118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:09.720207 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:09.720324 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:09.722070 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:09.722936 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:09.725761 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:09.726321 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:09.726416 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:09.730693 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:09.730698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:09.730701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:09.730732 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:09.733216 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:09.733988 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:09.733998 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:09.734004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:09.734151 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:09.734577 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:09.734585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:09.734592 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:09.734973 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:09.735125 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:09.735728 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 22:56:09.735980 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":242,"threads_started":1}
I20260927 22:56:09.737282 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:09.717442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44321 (local address 127.0.97.1:54087)
0927 22:56:09.718132 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:09.718173 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:09.719840 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:09.720586 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:09.720598 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:09.721220 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:09.721895 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:09.721916 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:09.723132 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:09.723138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:09.725508 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:09.725528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:09.726557 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:09.726567 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:09.726904 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:09.727612 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:09.727677 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:09.730468 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:09.733350 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:09.733358 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:09.733374 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:09.733856 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:09.734262 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:09.734266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:09.734268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:09.734454 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:09.735089 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:09.735095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:09.735375 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 22:56:09.735634 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 22:56:09.736173 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":279,"threads_started":1}
I20260927 22:56:09.738826 19907 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44321
I20260927 22:56:09.739137 19907 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:09.739949 19907 heartbeater.cc:511] Master 127.0.97.62:44321 requested a full tablet report, sending...
I20260927 22:56:09.742192 19718 ts_manager.cc:194] Registered new tserver with Master: 2ee789601fc642a8befc69a64f3b8f7d (127.0.97.1:32929)
I20260927 22:56:09.743551 19718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54087
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 22:56:09.777019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44321
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:10.090236 19913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:10.090492 19913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:10.090668 19913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:10.097838 19913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:10.097927 19913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:10.098088 19913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:56:10.107352 19913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:10.108439 19913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:10.109647 19913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:10.122237 19919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:10.122349 19918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:10.122203 19921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:10.123224 19913 server_base.cc:1034] running on GCE node
I20260927 22:56:10.123775 19913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:10.124665 19913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:10.125859 19913 hybrid_clock.cc:648] HybridClock initialized: now 1790549770125808 us; error 42 us; skew 500 ppm
I20260927 22:56:10.126157 19913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:10 dist-test-slave-df53 krb5kdc[19673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:10.129446 19913 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 22:56:10.130770 19913 webserver.cc:460] Webserver started at http://127.0.97.2:33321/ using document root <none> and password file <none>
I20260927 22:56:10.131417 19913 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:10.131548 19913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:10.131842 19913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:10.134596 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "3afad8b5b23e4d739ff560c5255eb93d"
format_stamp: "Formatted at 2026-09-27 22:56:10 on dist-test-slave-df53"
server_key: "f2f3f8670e5b4baed4139617ff2b9664"
server_key_iv: "9ddf4fd5ff2ec7003b110bb1cd9e8c62"
server_key_version: "encryptionkey@0"
I20260927 22:56:10.135335 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3afad8b5b23e4d739ff560c5255eb93d"
format_stamp: "Formatted at 2026-09-27 22:56:10 on dist-test-slave-df53"
server_key: "f2f3f8670e5b4baed4139617ff2b9664"
server_key_iv: "9ddf4fd5ff2ec7003b110bb1cd9e8c62"
server_key_version: "encryptionkey@0"
I20260927 22:56:10.139946 19913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:56:10.143441 19928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:10.144708 19913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:56:10.144876 19913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "3afad8b5b23e4d739ff560c5255eb93d"
format_stamp: "Formatted at 2026-09-27 22:56:10 on dist-test-slave-df53"
server_key: "f2f3f8670e5b4baed4139617ff2b9664"
server_key_iv: "9ddf4fd5ff2ec7003b110bb1cd9e8c62"
server_key_version: "encryptionkey@0"
I20260927 22:56:10.145059 19913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:10.167657 19913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:10.171729 19913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:10.172011 19913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:10.172894 19913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:10.174147 19913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:10.174258 19913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:10.174398 19913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:10.174494 19913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:10.206285 19913 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44303
I20260927 22:56:10.206368 20040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44303 every 8 connection(s)
I20260927 22:56:10.207458 19913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:56:10.208530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 19913
I20260927 22:56:10.208694 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:56:10.209197 388 external_mini_cluster.cc:1428] Setting key d8d9d24d24716184fe39bc3dd501bc4e
Sep 27 22:56:10 dist-test-slave-df53 krb5kdc[19673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:10.230196 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.210034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41123 (local address 127.0.97.62:44321)
0927 22:56:10.210249 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:10.210255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:10.211586 (+ 1331us) server_negotiation.cc:407] Connection header received
0927 22:56:10.214438 (+ 2852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:10.214442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:10.214508 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:10.214587 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:10.216446 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.217166 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.219737 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.220138 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.220233 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.223831 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:10.223837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:10.223840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:10.223871 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:10.227044 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:10.227756 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:10.227762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:10.227765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:10.227942 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:10.228459 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:10.228463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:10.228465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:10.228722 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:10.228829 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:10.229794 (+ 965us) server_negotiation.cc:299] Negotiation successful
0927 22:56:10.229969 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:56:10.231600 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.210613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44321 (local address 127.0.97.2:41123)
0927 22:56:10.211384 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:10.211451 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:10.214164 (+ 2713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:10.214803 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:10.214815 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:10.215554 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:10.216287 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:10.216311 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.217335 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.217343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:10.219498 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:10.219515 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.220357 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.220366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:10.220658 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.221328 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:10.221349 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:10.223601 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:10.227181 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:10.227189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:10.227204 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:10.227627 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:10.228083 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:10.228091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:10.228097 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:10.228327 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:10.228854 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:10.228864 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:10.229285 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 22:56:10.229584 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 22:56:10.230162 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":293,"threads_started":1}
I20260927 22:56:10.233064 20041 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44321
I20260927 22:56:10.233361 20041 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:10.234092 20041 heartbeater.cc:511] Master 127.0.97.62:44321 requested a full tablet report, sending...
I20260927 22:56:10.235487 19718 ts_manager.cc:194] Registered new tserver with Master: 3afad8b5b23e4d739ff560c5255eb93d (127.0.97.2:44303)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 22:56:10.236346 19718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 22:56:10.265059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44321
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:10.579167 20046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:10.579483 20046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:10.579654 20046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:10.587023 20046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:10.587159 20046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:10.587333 20046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:56:10.596745 20046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:10.598011 20046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:10.599261 20046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:10.612790 20051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:10.612737 20052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:10.612702 20054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:10.613252 20046 server_base.cc:1034] running on GCE node
I20260927 22:56:10.614042 20046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:10.614933 20046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:10.616174 20046 hybrid_clock.cc:648] HybridClock initialized: now 1790549770616141 us; error 55 us; skew 500 ppm
I20260927 22:56:10.616468 20046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:10 dist-test-slave-df53 krb5kdc[19673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:10.619472 20046 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 22:56:10.620795 20046 webserver.cc:460] Webserver started at http://127.0.97.3:41821/ using document root <none> and password file <none>
I20260927 22:56:10.621455 20046 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:10.621640 20046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:10.621963 20046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:10.624660 20046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "912ca39576af4ed0afcca2969f151931"
format_stamp: "Formatted at 2026-09-27 22:56:10 on dist-test-slave-df53"
server_key: "fa6c726e3228a249bb08b880f83e5b5c"
server_key_iv: "301116a6eb2ef2f2258033133cd9d8e6"
server_key_version: "encryptionkey@0"
I20260927 22:56:10.625389 20046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "912ca39576af4ed0afcca2969f151931"
format_stamp: "Formatted at 2026-09-27 22:56:10 on dist-test-slave-df53"
server_key: "fa6c726e3228a249bb08b880f83e5b5c"
server_key_iv: "301116a6eb2ef2f2258033133cd9d8e6"
server_key_version: "encryptionkey@0"
I20260927 22:56:10.630313 20046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:56:10.633525 20061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:10.634824 20046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:56:10.634999 20046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "912ca39576af4ed0afcca2969f151931"
format_stamp: "Formatted at 2026-09-27 22:56:10 on dist-test-slave-df53"
server_key: "fa6c726e3228a249bb08b880f83e5b5c"
server_key_iv: "301116a6eb2ef2f2258033133cd9d8e6"
server_key_version: "encryptionkey@0"
I20260927 22:56:10.635180 20046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:10.653002 20046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:10.656886 20046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:10.657168 20046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:10.658295 20046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:10.659586 20046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:10.659685 20046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:10.659826 20046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:10.659896 20046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:10.693001 20046 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38719
I20260927 22:56:10.693080 20173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38719 every 8 connection(s)
I20260927 22:56:10.694255 20046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:56:10.695451 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 20046
I20260927 22:56:10.695621 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:56:10.696178 388 external_mini_cluster.cc:1428] Setting key d046584418028863912292aad2147176
Sep 27 22:56:10 dist-test-slave-df53 krb5kdc[19673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:10.715016 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.697023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59547 (local address 127.0.97.62:44321)
0927 22:56:10.697248 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:10.697255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:10.698635 (+ 1380us) server_negotiation.cc:407] Connection header received
0927 22:56:10.700579 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:10.700585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:10.700671 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:10.700776 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:10.702418 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.703232 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.704735 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.705117 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.705246 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.708726 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:10.708731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:10.708735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:10.708773 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:10.711882 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:10.712595 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:10.712600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:10.712603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:10.712770 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:10.713255 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:10.713261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:10.713264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:10.713680 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:10.713825 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:10.714484 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 22:56:10.714688 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:56:10.715669 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.697665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44321 (local address 127.0.97.3:59547)
0927 22:56:10.698496 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:10.698530 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:10.700325 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:10.701009 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:10.701017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:10.701512 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:10.702235 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:10.702262 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.703381 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.703387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:10.704577 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:10.704588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.705342 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.705349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:10.705681 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.706277 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:10.706303 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:10.708518 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:10.712005 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:10.712011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:10.712022 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:10.712454 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:10.712871 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:10.712895 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:10.712900 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:10.713115 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:10.713798 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:10.713803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:10.714085 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 22:56:10.714316 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 22:56:10.714664 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":349,"threads_started":1}
I20260927 22:56:10.717118 20174 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44321
I20260927 22:56:10.717468 20174 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:10.718200 20174 heartbeater.cc:511] Master 127.0.97.62:44321 requested a full tablet report, sending...
I20260927 22:56:10.719736 19718 ts_manager.cc:194] Registered new tserver with Master: 912ca39576af4ed0afcca2969f151931 (127.0.97.3:38719)
I20260927 22:56:10.720559 19718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59547
I20260927 22:56:10.723557 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:56:10.740413 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.727396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.0.97.62:44321)
0927 22:56:10.727600 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:10.727608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:10.728029 (+ 421us) server_negotiation.cc:407] Connection header received
0927 22:56:10.728204 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:10.728210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:10.728275 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:10.728349 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:10.729805 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.730523 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.731791 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.732174 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.732265 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.735089 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:10.735093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:10.735097 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:10.735124 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:10.737719 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:10.738376 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:10.738383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:10.738387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:10.738544 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:10.738945 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:10.738950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:10.738960 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:10.739314 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:10.739551 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:10.739893 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 22:56:10.740114 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:56:10.746317 19907 heartbeater.cc:503] Master 127.0.97.62:44321 was elected leader, sending a full tablet report...
I20260927 22:56:10.743806 19718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:56:10.750888 19718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:56:10.772881 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.763833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.0.97.3:38719)
0927 22:56:10.765053 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:10.765061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:10.765092 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:56:10.765186 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:10.765194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:10.765428 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:10.766213 (+ 785us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:10.766401 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.767390 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.770262 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.771723 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.771859 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.772089 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:10.772376 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 22:56:10.772547 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":902,"threads_started":1}
I20260927 22:56:10.772970 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.763645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38719 (local address 127.0.0.1:51970)
0927 22:56:10.764291 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:10.764319 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:10.764475 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:10.765678 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:10.765683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:10.765707 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:10.766060 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:10.766068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.767552 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.767561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:10.770129 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:10.770139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.771912 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.771918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:10.772108 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.772177 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 22:56:10.772319 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:56:10.772500 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":1,"thread_start_us":322,"threads_started":1}
I20260927 22:56:10.773157 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.762636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44303 (local address 127.0.0.1:41422)
0927 22:56:10.763262 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:10.763305 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:10.763498 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:10.764287 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:10.764295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:10.764325 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:10.764735 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:10.764755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.767448 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.767454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:10.770135 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:10.770150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.772226 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.772234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:10.772424 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.772465 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:56:10.772553 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:56:10.772687 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":189,"thread_start_us":288,"threads_started":1}
I20260927 22:56:10.773865 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.762767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.0.97.2:44303)
0927 22:56:10.763283 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:10.763293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:10.763380 (+ 87us) server_negotiation.cc:407] Connection header received
0927 22:56:10.763763 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:10.763772 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:10.764046 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:10.765136 (+ 1090us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:10.765356 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.767287 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.770332 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.771734 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.771859 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.772059 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:10.773355 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0927 22:56:10.773566 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":283,"threads_started":1}
I20260927 22:56:10.776430 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.764793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32929 (local address 127.0.0.1:45490)
0927 22:56:10.768432 (+ 3639us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:10.768453 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:10.768572 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:10.769478 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:10.769482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:10.769505 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:10.769909 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:10.769920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.771351 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.771359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:10.774306 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:10.774322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.775716 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.775721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:10.775885 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.775919 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:56:10.776001 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 22:56:10.776120 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3489,"thread_start_us":3332,"threads_started":1}
I20260927 22:56:10.776499 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:10.765240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.0.97.1:32929)
0927 22:56:10.768014 (+ 2774us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:10.768023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:10.768709 (+ 686us) server_negotiation.cc:407] Connection header received
0927 22:56:10.768811 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:10.768820 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:10.769057 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:10.769235 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:10.770098 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.771200 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.774493 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:10.775516 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:10.775670 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:10.775897 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:10.776052 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 22:56:10.776208 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2582,"thread_start_us":2449,"threads_started":1}
I20260927 22:56:10.777493 19976 tablet_service.cc:1467] Processing CreateTablet for tablet 48ba882cc4644ed495064a69d97ea8ba (DEFAULT_TABLE table=test-table [id=4c226ff61a9f47fdafd14f056147b98b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:10.777444 20109 tablet_service.cc:1467] Processing CreateTablet for tablet 48ba882cc4644ed495064a69d97ea8ba (DEFAULT_TABLE table=test-table [id=4c226ff61a9f47fdafd14f056147b98b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:10.779106 19976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ba882cc4644ed495064a69d97ea8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:10.779632 20109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ba882cc4644ed495064a69d97ea8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:10.779867 19842 tablet_service.cc:1467] Processing CreateTablet for tablet 48ba882cc4644ed495064a69d97ea8ba (DEFAULT_TABLE table=test-table [id=4c226ff61a9f47fdafd14f056147b98b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:10.780848 19842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ba882cc4644ed495064a69d97ea8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:10.791270 20192 tablet_bootstrap.cc:492] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: Bootstrap starting.
I20260927 22:56:10.793469 20193 tablet_bootstrap.cc:492] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: Bootstrap starting.
I20260927 22:56:10.794945 20192 tablet_bootstrap.cc:654] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:10.795199 20194 tablet_bootstrap.cc:492] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: Bootstrap starting.
I20260927 22:56:10.796293 20192 log.cc:824] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:10.796878 20193 tablet_bootstrap.cc:654] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:10.797992 20193 log.cc:824] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:10.798455 20194 tablet_bootstrap.cc:654] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:10.799356 20192 tablet_bootstrap.cc:492] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: No bootstrap required, opened a new log
I20260927 22:56:10.799685 20192 ts_tablet_manager.cc:1397] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 22:56:10.799670 20194 log.cc:824] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:10.801074 20193 tablet_bootstrap.cc:492] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: No bootstrap required, opened a new log
I20260927 22:56:10.801379 20193 ts_tablet_manager.cc:1397] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:56:10.807395 20192 raft_consensus.cc:348] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afad8b5b23e4d739ff560c5255eb93d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } } peers { permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } } peers { permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 } }
I20260927 22:56:10.807766 20192 raft_consensus.cc:374] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:10.807871 20192 raft_consensus.cc:729] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee789601fc642a8befc69a64f3b8f7d, State: Initialized, Role: FOLLOWER
I20260927 22:56:10.808478 20192 consensus_queue.cc:260] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afad8b5b23e4d739ff560c5255eb93d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } } peers { permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } } peers { permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 } }
I20260927 22:56:10.808852 20194 tablet_bootstrap.cc:492] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: No bootstrap required, opened a new log
I20260927 22:56:10.809172 20194 ts_tablet_manager.cc:1397] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260927 22:56:10.810233 20192 ts_tablet_manager.cc:1428] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260927 22:56:10.812677 20193 raft_consensus.cc:348] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afad8b5b23e4d739ff560c5255eb93d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } } peers { permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } } peers { permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 } }
I20260927 22:56:10.813122 20193 raft_consensus.cc:374] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:10.813226 20193 raft_consensus.cc:729] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afad8b5b23e4d739ff560c5255eb93d, State: Initialized, Role: FOLLOWER
I20260927 22:56:10.813845 20193 consensus_queue.cc:260] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afad8b5b23e4d739ff560c5255eb93d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } } peers { permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } } peers { permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 } }
I20260927 22:56:10.815120 20041 heartbeater.cc:503] Master 127.0.97.62:44321 was elected leader, sending a full tablet report...
I20260927 22:56:10.815555 20193 ts_tablet_manager.cc:1428] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 22:56:10.819900 20194 raft_consensus.cc:348] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afad8b5b23e4d739ff560c5255eb93d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } } peers { permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } } peers { permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 } }
I20260927 22:56:10.820228 20194 raft_consensus.cc:374] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:10.820338 20194 raft_consensus.cc:729] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912ca39576af4ed0afcca2969f151931, State: Initialized, Role: FOLLOWER
I20260927 22:56:10.820897 20194 consensus_queue.cc:260] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afad8b5b23e4d739ff560c5255eb93d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } } peers { permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } } peers { permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 } }
I20260927 22:56:10.821925 20174 heartbeater.cc:503] Master 127.0.97.62:44321 was elected leader, sending a full tablet report...
I20260927 22:56:10.822480 20194 ts_tablet_manager.cc:1428] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260927 22:56:10.945902 20175 tablet.cc:2369] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:56:10.959358 20042 tablet.cc:2369] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:56:10.967016 19908 tablet.cc:2369] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:11.062889 20199 raft_consensus.cc:482] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:56:11.063210 20199 raft_consensus.cc:504] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afad8b5b23e4d739ff560c5255eb93d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } } peers { permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } } peers { permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 } }
I20260927 22:56:11.064817 20199 leader_election.cc:288] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912ca39576af4ed0afcca2969f151931 (127.0.97.3:38719), 2ee789601fc642a8befc69a64f3b8f7d (127.0.97.1:32929)
I20260927 22:56:11.072118 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.065425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50431 (local address 127.0.97.3:38719)
0927 22:56:11.065652 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:11.065658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:11.066231 (+ 573us) server_negotiation.cc:407] Connection header received
0927 22:56:11.066462 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:11.066466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:11.066546 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:11.066642 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:11.067332 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.068300 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.070254 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.071187 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.071279 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:11.071364 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:11.071749 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 22:56:11.071856 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:56:11.072220 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.065329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38719 (local address 127.0.97.2:50431)
0927 22:56:11.066114 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.066143 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:11.066286 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:11.066728 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:11.066733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:11.066769 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:11.067138 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:11.067150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.068423 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.068428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:11.070087 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:11.070099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.071401 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.071410 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:11.071588 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:11.071626 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:56:11.071729 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 22:56:11.071872 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":429,"threads_started":1}
I20260927 22:56:11.072817 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.066445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37169 (local address 127.0.97.1:32929)
0927 22:56:11.066639 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:11.066644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:11.067007 (+ 363us) server_negotiation.cc:407] Connection header received
0927 22:56:11.067132 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:11.067137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:11.067218 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:11.067310 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:11.067910 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.068713 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.070742 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.071774 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.071887 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:11.071982 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:11.072450 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 22:56:11.072586 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:56:11.072849 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.066228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32929 (local address 127.0.97.2:37169)
0927 22:56:11.066893 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.066911 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:11.067013 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:11.067352 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:11.067356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:11.067383 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:11.067763 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:11.067774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.068861 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.068865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:11.070574 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:11.070588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.072011 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.072018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:11.072219 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:11.072250 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:56:11.072414 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 22:56:11.072532 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":371,"threads_started":1}
I20260927 22:56:11.073789 20129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ba882cc4644ed495064a69d97ea8ba" candidate_uuid: "3afad8b5b23e4d739ff560c5255eb93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912ca39576af4ed0afcca2969f151931" is_pre_election: true
I20260927 22:56:11.073872 19862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ba882cc4644ed495064a69d97ea8ba" candidate_uuid: "3afad8b5b23e4d739ff560c5255eb93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee789601fc642a8befc69a64f3b8f7d" is_pre_election: true
I20260927 22:56:11.074298 20129 raft_consensus.cc:2454] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3afad8b5b23e4d739ff560c5255eb93d in term 0.
I20260927 22:56:11.074370 19862 raft_consensus.cc:2454] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3afad8b5b23e4d739ff560c5255eb93d in term 0.
I20260927 22:56:11.074978 19930 leader_election.cc:302] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3afad8b5b23e4d739ff560c5255eb93d, 912ca39576af4ed0afcca2969f151931; no voters:
I20260927 22:56:11.075304 20199 raft_consensus.cc:2781] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:56:11.075448 20199 raft_consensus.cc:482] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:56:11.075551 20199 raft_consensus.cc:3037] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:11.077699 20199 raft_consensus.cc:504] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afad8b5b23e4d739ff560c5255eb93d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } } peers { permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } } peers { permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 } }
I20260927 22:56:11.078289 20199 leader_election.cc:288] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [CANDIDATE]: Term 1 election: Requested vote from peers 912ca39576af4ed0afcca2969f151931 (127.0.97.3:38719), 2ee789601fc642a8befc69a64f3b8f7d (127.0.97.1:32929)
I20260927 22:56:11.078670 20129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ba882cc4644ed495064a69d97ea8ba" candidate_uuid: "3afad8b5b23e4d739ff560c5255eb93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912ca39576af4ed0afcca2969f151931"
I20260927 22:56:11.078814 19862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ba882cc4644ed495064a69d97ea8ba" candidate_uuid: "3afad8b5b23e4d739ff560c5255eb93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee789601fc642a8befc69a64f3b8f7d"
I20260927 22:56:11.078866 20129 raft_consensus.cc:3037] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:11.079011 19862 raft_consensus.cc:3037] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:11.080947 20129 raft_consensus.cc:2454] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3afad8b5b23e4d739ff560c5255eb93d in term 1.
I20260927 22:56:11.081158 19862 raft_consensus.cc:2454] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3afad8b5b23e4d739ff560c5255eb93d in term 1.
I20260927 22:56:11.081328 19930 leader_election.cc:302] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3afad8b5b23e4d739ff560c5255eb93d, 912ca39576af4ed0afcca2969f151931; no voters:
I20260927 22:56:11.081653 20199 raft_consensus.cc:2781] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:11.082069 20199 raft_consensus.cc:686] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 1 LEADER]: Becoming Leader. State: Replica: 3afad8b5b23e4d739ff560c5255eb93d, State: Running, Role: LEADER
I20260927 22:56:11.082772 20199 consensus_queue.cc:237] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afad8b5b23e4d739ff560c5255eb93d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } } peers { permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } } peers { permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 } }
I20260927 22:56:11.087752 19717 catalog_manager.cc:5495] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3afad8b5b23e4d739ff560c5255eb93d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3afad8b5b23e4d739ff560c5255eb93d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3afad8b5b23e4d739ff560c5255eb93d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 } health_report { overall_health: UNKNOWN } } }
I20260927 22:56:11.135097 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.129174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.0.97.2:44303)
0927 22:56:11.129406 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:11.129412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:11.129433 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:56:11.129638 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:11.129642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:11.129735 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:11.129932 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:11.130328 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.131145 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.132718 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.133195 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.133348 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:11.133963 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:11.134466 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:11.134636 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:11.134699 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 22:56:11.134781 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:56:11.143253 19862 raft_consensus.cc:1261] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d [term 1 FOLLOWER]: Refusing update from remote peer 3afad8b5b23e4d739ff560c5255eb93d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:11.143250 20129 raft_consensus.cc:1261] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931 [term 1 FOLLOWER]: Refusing update from remote peer 3afad8b5b23e4d739ff560c5255eb93d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:11.144228 20203 consensus_queue.cc:1035] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ee789601fc642a8befc69a64f3b8f7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:11.144594 20199 consensus_queue.cc:1035] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "912ca39576af4ed0afcca2969f151931" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:11.164772 20206 mvcc.cc:205] Tried to move back new op lower bound from 7334091862592790528 to 7334091862360023040. Current Snapshot: MvccSnapshot[applied={T|T < 7334091862592790528}]
I20260927 22:56:11.182377 19718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:11.182687 19718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:11.185943 19718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b493db9a9d64ee0896c219ae5ba8ddd: Sending DeleteTablet for 3 replicas of tablet 48ba882cc4644ed495064a69d97ea8ba
I20260927 22:56:11.186992 19976 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ba882cc4644ed495064a69d97ea8ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41422
I20260927 22:56:11.187103 20109 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ba882cc4644ed495064a69d97ea8ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51970
I20260927 22:56:11.187211 19842 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ba882cc4644ed495064a69d97ea8ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45490
I20260927 22:56:11.188927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 19779
I20260927 22:56:11.203382 20217 tablet_replica.cc:330] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: stopping tablet replica
I20260927 22:56:11.203933 20217 raft_consensus.cc:2229] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:11.204555 20217 raft_consensus.cc:2258] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:11.205399 20218 tablet_replica.cc:330] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: stopping tablet replica
I20260927 22:56:11.205809 20218 raft_consensus.cc:2229] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:11.206420 20218 raft_consensus.cc:2258] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:11.207037 20217 ts_tablet_manager.cc:1905] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:11.213364 20217 ts_tablet_manager.cc:1918] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:11.213579 20217 log.cc:1196] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/48ba882cc4644ed495064a69d97ea8ba
I20260927 22:56:11.214143 20217 ts_tablet_manager.cc:1939] T 48ba882cc4644ed495064a69d97ea8ba P 912ca39576af4ed0afcca2969f151931: Deleting consensus metadata
I20260927 22:56:11.215948 19703 catalog_manager.cc:4841] TS 912ca39576af4ed0afcca2969f151931 (127.0.97.3:38719): tablet 48ba882cc4644ed495064a69d97ea8ba (table test-table [id=4c226ff61a9f47fdafd14f056147b98b]) successfully deleted
I20260927 22:56:11.218034 20218 ts_tablet_manager.cc:1905] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:11.224795 20218 ts_tablet_manager.cc:1918] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:11.227097 20216 tablet_replica.cc:330] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: stopping tablet replica
I20260927 22:56:11.227638 20216 raft_consensus.cc:2229] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:56:11.228408 20216 raft_consensus.cc:2258] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 22:56:11 dist-test-slave-df53 krb5kdc[19673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 22:56:11.235553 20216 ts_tablet_manager.cc:1905] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:11.235598 20218 log.cc:1196] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-0/wal/wals/48ba882cc4644ed495064a69d97ea8ba
I20260927 22:56:11.236164 20218 ts_tablet_manager.cc:1939] T 48ba882cc4644ed495064a69d97ea8ba P 2ee789601fc642a8befc69a64f3b8f7d: Deleting consensus metadata
I20260927 22:56:11.242638 19703 catalog_manager.cc:4841] TS 2ee789601fc642a8befc69a64f3b8f7d (127.0.97.1:32929): tablet 48ba882cc4644ed495064a69d97ea8ba (table test-table [id=4c226ff61a9f47fdafd14f056147b98b]) successfully deleted
I20260927 22:56:11.243162 20216 ts_tablet_manager.cc:1918] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:11.243323 20216 log.cc:1196] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/48ba882cc4644ed495064a69d97ea8ba
I20260927 22:56:11.243768 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.190320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.0.97.1:32929)
0927 22:56:11.190532 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:11.190540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:11.212375 (+ 21835us) server_negotiation.cc:407] Connection header received
0927 22:56:11.212815 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:11.212821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:11.212906 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:11.212973 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:11.214840 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.215934 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.228698 (+ 12764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.229543 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.229666 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:11.236398 (+ 6732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:11.236414 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:11.236430 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:11.236481 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:11.240127 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:11.240990 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:11.240998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:11.241001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:11.241162 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:11.241697 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:11.241702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:11.241705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:11.242161 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:11.242446 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:11.243090 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 22:56:11.243320 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:56:11.243873 20216 ts_tablet_manager.cc:1939] T 48ba882cc4644ed495064a69d97ea8ba P 3afad8b5b23e4d739ff560c5255eb93d: Deleting consensus metadata
I20260927 22:56:11.244894 19902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:11.245455 19705 catalog_manager.cc:4841] TS 3afad8b5b23e4d739ff560c5255eb93d (127.0.97.2:44303): tablet 48ba882cc4644ed495064a69d97ea8ba (table test-table [id=4c226ff61a9f47fdafd14f056147b98b]) successfully deleted
I20260927 22:56:11.334422 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 19779
I20260927 22:56:11.352761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 19913
Sep 27 22:56:11 dist-test-slave-df53 krb5kdc[19673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 22:56:11.369118 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.353547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.0.97.2:44303)
0927 22:56:11.353778 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:11.353785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:11.353807 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:56:11.353943 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:11.353949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:11.354038 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:11.354156 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:11.355350 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.356550 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.357877 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.358563 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.358696 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:11.362676 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:11.362690 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:11.362708 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:11.362747 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:11.366437 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:11.367076 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:11.367085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:11.367089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:11.367262 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:11.367589 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:11.367594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:11.367597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:11.368008 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:11.368331 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:11.368491 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 22:56:11.368694 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 22:56:11.369894 20036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:11.483343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 19913
I20260927 22:56:11.502031 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 20046
Sep 27 22:56:11 dist-test-slave-df53 krb5kdc[19673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 22:56:11.516772 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.502888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.0.97.3:38719)
0927 22:56:11.503102 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:11.503107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:11.503258 (+ 151us) server_negotiation.cc:407] Connection header received
0927 22:56:11.503414 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:11.503418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:11.503485 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:11.503622 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:11.504887 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.505571 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.506902 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.507298 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.507389 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:11.511800 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:11.511813 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:11.511828 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:11.511859 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:11.514412 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:11.515038 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:11.515044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:11.515046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:11.515178 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:11.515598 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:11.515601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:11.515603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:11.515913 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:11.516130 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:11.516372 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 22:56:11.516513 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:56:11.517405 20169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:11.608572 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 20046
I20260927 22:56:11.624737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 19686
I20260927 22:56:11.625763 19748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:11.733749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 19686
2026-09-27T22:56:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4334 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T22:56:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:11Z Disabled control of system clock
I20260927 22:56:11.806609 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46329
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:56:12.116676 20227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:12.116951 20227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:12.117029 20227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:12.124101 20227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:12.124202 20227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:12.124239 20227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:12.124295 20227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:12.124339 20227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:12.133342 20227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:12.134534 20227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:12.136253 20227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:12.149473 20232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:12.149505 20235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:12.149573 20233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:12.150233 20227 server_base.cc:1034] running on GCE node
I20260927 22:56:12.150986 20227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:12.152316 20227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:12.153535 20227 hybrid_clock.cc:648] HybridClock initialized: now 1790549772153496 us; error 55 us; skew 500 ppm
I20260927 22:56:12.153872 20227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:56:12.156055 20227 webserver.cc:460] Webserver started at http://127.0.97.62:43251/ using document root <none> and password file <none>
I20260927 22:56:12.156709 20227 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:12.156831 20227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:12.157132 20227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:12.159871 20227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "72e9f6c43bb14aa0b8fb979973318ee2"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-df53"
server_key: "0d2ebc6c81f26f37cd29ddd28793fcd8"
server_key_iv: "bc50a34b9ae818df631d196833fa74cb"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.160647 20227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "72e9f6c43bb14aa0b8fb979973318ee2"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-df53"
server_key: "0d2ebc6c81f26f37cd29ddd28793fcd8"
server_key_iv: "bc50a34b9ae818df631d196833fa74cb"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.165465 20227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 22:56:12.168778 20241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:12.170094 20227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:12.170270 20227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "72e9f6c43bb14aa0b8fb979973318ee2"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-df53"
server_key: "0d2ebc6c81f26f37cd29ddd28793fcd8"
server_key_iv: "bc50a34b9ae818df631d196833fa74cb"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.170477 20227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:12.195220 20227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:12.196081 20227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:12.196372 20227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:12.214756 20227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46329
I20260927 22:56:12.214774 20292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46329 every 8 connection(s)
I20260927 22:56:12.216044 20227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:12.219652 20293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:12.226421 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 20227
I20260927 22:56:12.226604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:12.227180 388 external_mini_cluster.cc:1428] Setting key 27049646abd8451de703f7f8adb9d6f2
I20260927 22:56:12.230587 20293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2: Bootstrap starting.
I20260927 22:56:12.233739 20293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:12.234761 20293 log.cc:824] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:12.237315 20293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2: No bootstrap required, opened a new log
I20260927 22:56:12.240429 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.229111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.0.97.62:46329)
0927 22:56:12.230106 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.230117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.230157 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:56:12.232061 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.232086 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.232528 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.232910 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.233455 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.234910 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.236228 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.236904 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.237110 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.238296 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.238303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.238316 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:12.238823 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.238904 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:12.238931 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.239065 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.239256 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.239540 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":321,"threads_started":1}
I20260927 22:56:12.245221 20293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e9f6c43bb14aa0b8fb979973318ee2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46329 } }
I20260927 22:56:12.245568 20293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:12.245735 20293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e9f6c43bb14aa0b8fb979973318ee2, State: Initialized, Role: FOLLOWER
I20260927 22:56:12.246297 20293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e9f6c43bb14aa0b8fb979973318ee2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46329 } }
I20260927 22:56:12.246531 20293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:12.246673 20293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:12.246881 20293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:12.249024 20293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e9f6c43bb14aa0b8fb979973318ee2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46329 } }
I20260927 22:56:12.249501 20293 leader_election.cc:302] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72e9f6c43bb14aa0b8fb979973318ee2; no voters:
I20260927 22:56:12.250098 20293 leader_election.cc:288] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:12.250250 20298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:12.250953 20298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [term 1 LEADER]: Becoming Leader. State: Replica: 72e9f6c43bb14aa0b8fb979973318ee2, State: Running, Role: LEADER
I20260927 22:56:12.251533 20298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e9f6c43bb14aa0b8fb979973318ee2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46329 } }
I20260927 22:56:12.252015 20293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:12.254652 20300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72e9f6c43bb14aa0b8fb979973318ee2. Latest consensus state: current_term: 1 leader_uuid: "72e9f6c43bb14aa0b8fb979973318ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e9f6c43bb14aa0b8fb979973318ee2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46329 } } }
I20260927 22:56:12.254734 20299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72e9f6c43bb14aa0b8fb979973318ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e9f6c43bb14aa0b8fb979973318ee2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46329 } } }
I20260927 22:56:12.255189 20300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:12.255199 20299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:12.258589 20306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:12.263137 20306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:12.271303 20306 catalog_manager.cc:1347] Generated new cluster ID: 24ad1cfdab404df0acc6b6b7d1511b00
I20260927 22:56:12.271461 20306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:12.298825 20306 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:12.300624 20306 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:12.308919 20306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2: Generated new TSK 0
I20260927 22:56:12.309894 20306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:12.325842 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46329
--builtin_ntp_servers=127.0.97.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:56:12.650815 20317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:12.651115 20317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:12.651273 20317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:12.658433 20317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:12.658551 20317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:12.658720 20317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 22:56:12.667536 20317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:12.668752 20317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:12.670518 20317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:12.683589 20322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:12.683558 20325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:12.683557 20323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:12.684449 20317 server_base.cc:1034] running on GCE node
I20260927 22:56:12.684963 20317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:12.685853 20317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:12.687073 20317 hybrid_clock.cc:648] HybridClock initialized: now 1790549772687037 us; error 27 us; skew 500 ppm
I20260927 22:56:12.687338 20317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:56:12.689496 20317 webserver.cc:460] Webserver started at http://127.0.97.1:33267/ using document root <none> and password file <none>
I20260927 22:56:12.690231 20317 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:12.690317 20317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:12.690579 20317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:12.693209 20317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/data/instance:
uuid: "67ca388559434c3fa58d4957d5509785"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-df53"
server_key: "a04cf6e7b643e2df24529a190f3e92c3"
server_key_iv: "b03abd705d14341abdce5e6199dedebc"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.693980 20317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/wal/instance:
uuid: "67ca388559434c3fa58d4957d5509785"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-df53"
server_key: "a04cf6e7b643e2df24529a190f3e92c3"
server_key_iv: "b03abd705d14341abdce5e6199dedebc"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.699074 20317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:56:12.702440 20331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:12.703621 20317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:56:12.703738 20317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/wal
uuid: "67ca388559434c3fa58d4957d5509785"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-df53"
server_key: "a04cf6e7b643e2df24529a190f3e92c3"
server_key_iv: "b03abd705d14341abdce5e6199dedebc"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.703899 20317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:12.724503 20317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:12.725258 20317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:12.725502 20317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:12.726365 20317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:12.727725 20317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:12.727808 20317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:12.727885 20317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:12.727941 20317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:12.761368 20317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38905
I20260927 22:56:12.761474 20443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38905 every 8 connection(s)
I20260927 22:56:12.762701 20317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/data/info.pb
I20260927 22:56:12.771525 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 20317
I20260927 22:56:12.771710 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-0/wal/instance
I20260927 22:56:12.772112 388 external_mini_cluster.cc:1428] Setting key 8a66dccd9c69c8f50e78b0332514b8e9
I20260927 22:56:12.774886 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46329
--builtin_ntp_servers=127.0.97.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:56:12.775442 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.765242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38053 (local address 127.0.97.62:46329)
0927 22:56:12.765705 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.765710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.766925 (+ 1215us) server_negotiation.cc:407] Connection header received
0927 22:56:12.768734 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.768740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.768836 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.768956 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.770070 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.770798 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.772487 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.773021 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.773156 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.774595 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.774600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.774603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:12.774620 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.774664 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:12.774676 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.774792 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.775038 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.775130 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":214,"threads_started":1}
I20260927 22:56:12.776394 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.765906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46329 (local address 127.0.97.1:38053)
0927 22:56:12.766772 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.766807 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:12.768472 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:12.769186 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:12.769195 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:12.769286 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:12.769898 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.769914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.770942 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.770954 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.772328 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.772340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.773259 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.773268 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.773509 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.774294 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:56:12.774297 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:12.774335 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:56:12.774410 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:12.774790 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:12.774802 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:12.774812 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:56:12.775017 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 22:56:12.775392 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":241,"threads_started":1}
I20260927 22:56:12.777908 20444 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46329
I20260927 22:56:12.778215 20444 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:12.779060 20444 heartbeater.cc:511] Master 127.0.97.62:46329 requested a full tablet report, sending...
I20260927 22:56:12.781266 20258 ts_manager.cc:194] Registered new tserver with Master: 67ca388559434c3fa58d4957d5509785 (127.0.97.1:38905)
I20260927 22:56:12.783363 20258 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:38053
W20260927 22:56:13.091771 20448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:13.092020 20448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:13.092154 20448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:13.099287 20448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:13.099373 20448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:13.099532 20448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 22:56:13.108505 20448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:13.109668 20448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:13.111308 20448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:13.123958 20456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:13.124085 20453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:13.124058 20454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:13.124694 20448 server_base.cc:1034] running on GCE node
I20260927 22:56:13.125303 20448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:13.126288 20448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:13.127489 20448 hybrid_clock.cc:648] HybridClock initialized: now 1790549773127426 us; error 53 us; skew 500 ppm
I20260927 22:56:13.127800 20448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:56:13.129966 20448 webserver.cc:460] Webserver started at http://127.0.97.2:40801/ using document root <none> and password file <none>
I20260927 22:56:13.130554 20448 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:13.130635 20448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:13.130892 20448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:13.133671 20448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/data/instance:
uuid: "1c37291dc9f04acf86561fd70f3e7450"
format_stamp: "Formatted at 2026-09-27 22:56:13 on dist-test-slave-df53"
server_key: "5fdc147eb843533b82ccf81d8daebc84"
server_key_iv: "ab35e1c457911ee2faa75920f4d907ee"
server_key_version: "encryptionkey@0"
I20260927 22:56:13.134441 20448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1c37291dc9f04acf86561fd70f3e7450"
format_stamp: "Formatted at 2026-09-27 22:56:13 on dist-test-slave-df53"
server_key: "5fdc147eb843533b82ccf81d8daebc84"
server_key_iv: "ab35e1c457911ee2faa75920f4d907ee"
server_key_version: "encryptionkey@0"
I20260927 22:56:13.139029 20448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:56:13.142333 20462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:13.143566 20448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:56:13.143724 20448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/wal
uuid: "1c37291dc9f04acf86561fd70f3e7450"
format_stamp: "Formatted at 2026-09-27 22:56:13 on dist-test-slave-df53"
server_key: "5fdc147eb843533b82ccf81d8daebc84"
server_key_iv: "ab35e1c457911ee2faa75920f4d907ee"
server_key_version: "encryptionkey@0"
I20260927 22:56:13.143986 20448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:13.188565 20448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:13.189683 20448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:13.189968 20448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:13.190822 20448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:13.192199 20448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:13.192282 20448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:13.192355 20448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:13.192417 20448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:13.225131 20448 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46835
I20260927 22:56:13.225217 20574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46835 every 8 connection(s)
I20260927 22:56:13.226415 20448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/data/info.pb
I20260927 22:56:13.229771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 20448
I20260927 22:56:13.229964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/wal/instance
I20260927 22:56:13.230475 388 external_mini_cluster.cc:1428] Setting key 75f63e5492697911a8e6d237a78496ae
I20260927 22:56:13.232982 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46329
--builtin_ntp_servers=127.0.97.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:56:13.239570 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.228905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38967 (local address 127.0.97.62:46329)
0927 22:56:13.229110 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:13.229116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:13.230665 (+ 1549us) server_negotiation.cc:407] Connection header received
0927 22:56:13.233298 (+ 2633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:13.233304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:13.233396 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:13.233507 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:13.234650 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.235347 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.236871 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.237265 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.237403 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.238797 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:13.238801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:13.238805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:13.238822 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:13.238852 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:13.238864 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:13.238962 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:13.239186 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 22:56:13.239279 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:56:13.240628 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.229458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46329 (local address 127.0.97.2:38967)
0927 22:56:13.230522 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:13.230564 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:13.232932 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:13.233756 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:13.233764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:13.233850 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:13.234484 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.234501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.235506 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.235511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.236713 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.236725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.237489 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.237496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.237780 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.238500 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:56:13.238502 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:13.238538 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:56:13.238609 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:13.238972 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:13.238984 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:13.238998 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:56:13.239157 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:56:13.239526 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":313,"threads_started":1}
I20260927 22:56:13.242043 20575 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46329
I20260927 22:56:13.242336 20575 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:13.243057 20575 heartbeater.cc:511] Master 127.0.97.62:46329 requested a full tablet report, sending...
I20260927 22:56:13.244380 20258 ts_manager.cc:194] Registered new tserver with Master: 1c37291dc9f04acf86561fd70f3e7450 (127.0.97.2:46835)
I20260927 22:56:13.245143 20258 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:38967
W20260927 22:56:13.550691 20578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:13.550972 20578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:13.551110 20578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:13.558420 20578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:13.558535 20578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:13.558696 20578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 22:56:13.567481 20578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:13.568666 20578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:13.570412 20578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:13.585106 20584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:13.585110 20583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:13.585228 20586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:13.590986 20578 server_base.cc:1034] running on GCE node
I20260927 22:56:13.591630 20578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:13.592547 20578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:13.593739 20578 hybrid_clock.cc:648] HybridClock initialized: now 1790549773593694 us; error 44 us; skew 500 ppm
I20260927 22:56:13.594049 20578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:56:13.601889 20578 webserver.cc:460] Webserver started at http://127.0.97.3:39305/ using document root <none> and password file <none>
I20260927 22:56:13.602578 20578 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:13.602690 20578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:13.603010 20578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:13.605875 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/data/instance:
uuid: "20169a683d314643832617d78f80960e"
format_stamp: "Formatted at 2026-09-27 22:56:13 on dist-test-slave-df53"
server_key: "f9dfcef78c2cc77dedd855e5301d20c2"
server_key_iv: "e36ac3271d02e5dd482964d70cbd7f73"
server_key_version: "encryptionkey@0"
I20260927 22:56:13.606630 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/wal/instance:
uuid: "20169a683d314643832617d78f80960e"
format_stamp: "Formatted at 2026-09-27 22:56:13 on dist-test-slave-df53"
server_key: "f9dfcef78c2cc77dedd855e5301d20c2"
server_key_iv: "e36ac3271d02e5dd482964d70cbd7f73"
server_key_version: "encryptionkey@0"
I20260927 22:56:13.611449 20578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 22:56:13.614802 20592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:13.616097 20578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 22:56:13.616250 20578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/wal
uuid: "20169a683d314643832617d78f80960e"
format_stamp: "Formatted at 2026-09-27 22:56:13 on dist-test-slave-df53"
server_key: "f9dfcef78c2cc77dedd855e5301d20c2"
server_key_iv: "e36ac3271d02e5dd482964d70cbd7f73"
server_key_version: "encryptionkey@0"
I20260927 22:56:13.616449 20578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:13.627882 20578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:13.628612 20578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:13.628870 20578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:13.629803 20578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:13.631127 20578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:13.631237 20578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:13.631376 20578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:13.631469 20578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:13.672775 20578 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33223
I20260927 22:56:13.672891 20704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33223 every 8 connection(s)
I20260927 22:56:13.674306 20578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/data/info.pb
I20260927 22:56:13.678925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 20578
I20260927 22:56:13.679103 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/wal/instance
I20260927 22:56:13.679675 388 external_mini_cluster.cc:1428] Setting key d3f5e4dda606ed57c7f27fcf1a370ae8
I20260927 22:56:13.689157 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.677350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58901 (local address 127.0.97.62:46329)
0927 22:56:13.677610 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:13.677619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:13.678960 (+ 1341us) server_negotiation.cc:407] Connection header received
0927 22:56:13.681574 (+ 2614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:13.681580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:13.681722 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:13.681848 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:13.683931 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.684752 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.686394 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.686821 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.686975 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.688319 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:13.688328 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:13.688333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:13.688354 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:13.688398 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:13.688413 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:13.688512 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:13.688729 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 22:56:13.688824 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 22:56:13.690347 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.677912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46329 (local address 127.0.97.3:58901)
0927 22:56:13.678798 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:13.678858 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:13.681226 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:13.683012 (+ 1786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:13.683024 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:13.683138 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:13.683758 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.683785 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.684901 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.684907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.686249 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.686261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.687059 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.687067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.687292 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.687988 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:56:13.687991 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:13.688030 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:56:13.688100 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:13.688511 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:13.688523 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:13.688533 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:56:13.688717 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 22:56:13.689104 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":268,"threads_started":1}
I20260927 22:56:13.694717 20705 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46329
I20260927 22:56:13.694985 20705 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:13.695734 20705 heartbeater.cc:511] Master 127.0.97.62:46329 requested a full tablet report, sending...
I20260927 22:56:13.697144 20257 ts_manager.cc:194] Registered new tserver with Master: 20169a683d314643832617d78f80960e (127.0.97.3:33223)
I20260927 22:56:13.698482 20257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:58901
I20260927 22:56:13.707639 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:56:13.717499 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.711643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.0.97.62:46329)
0927 22:56:13.711852 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:13.711860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:13.712252 (+ 392us) server_negotiation.cc:407] Connection header received
0927 22:56:13.712455 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:13.712459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:13.712530 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:13.712631 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:13.713113 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.713861 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.715231 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.715641 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.715767 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.716830 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:13.716834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:13.716838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:13.716854 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:13.716883 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:13.716894 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:13.716996 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:13.717137 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 22:56:13.717223 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:56:13.721138 20257 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:56:13.727651 20257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:56:13.749377 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.739147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46835 (local address 127.0.0.1:40096)
0927 22:56:13.740162 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:13.740196 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:13.740316 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:13.741504 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:13.741513 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:13.741541 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:13.741984 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.741998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.744013 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.744021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.746584 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.746598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.748439 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.748445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.748659 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.748736 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 22:56:13.748869 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 22:56:13.749044 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":713,"threads_started":1}
I20260927 22:56:13.749393 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.739467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.0.97.2:46835)
0927 22:56:13.740806 (+ 1339us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:13.740816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:13.740845 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:56:13.740964 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:13.740974 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:13.741241 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:13.742185 (+ 944us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:13.742377 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.743884 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.746822 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.748259 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.748399 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.748628 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:13.748914 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 22:56:13.749077 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1152,"thread_start_us":1034,"threads_started":1}
I20260927 22:56:13.750859 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.740736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38905 (local address 127.0.0.1:39984)
0927 22:56:13.741583 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:13.741622 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:13.741750 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:13.743692 (+ 1942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:13.743700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:13.743737 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:13.744257 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.744268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.746266 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.746271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.747909 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.747941 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.749981 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.749989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.750237 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.750272 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:56:13.750354 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 22:56:13.750467 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":29,"spinlock_wait_cycles":11520,"thread_start_us":584,"threads_started":1}
I20260927 22:56:13.751797 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.741051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.0.97.1:38905)
0927 22:56:13.743160 (+ 2109us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:13.743168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:13.743191 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:56:13.743272 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:13.743278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:13.743474 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:13.743647 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:13.744526 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.746117 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.748110 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.749757 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.750981 (+ 1224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.751200 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:13.751265 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 22:56:13.751441 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1958,"thread_start_us":1833,"threads_started":1}
I20260927 22:56:13.753019 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.742677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33223 (local address 127.0.0.1:54496)
0927 22:56:13.744442 (+ 1765us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:13.744458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:13.744569 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:13.745837 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:13.745845 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:13.745868 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:13.746299 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.746309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.748177 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.748182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.750334 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.750348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.752297 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.752303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.752470 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.752509 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:56:13.752600 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:56:13.752715 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1641,"mutex_wait_us":1,"thread_start_us":1484,"threads_started":1}
I20260927 22:56:13.753162 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.742202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.0.97.3:33223)
0927 22:56:13.745091 (+ 2889us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:13.745099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:13.745127 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:56:13.745211 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:13.745218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:13.745419 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:13.745608 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:13.746519 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.748064 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.750560 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.752116 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.752268 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.752532 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:13.752648 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 22:56:13.752791 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2723,"thread_start_us":2573,"threads_started":1}
I20260927 22:56:13.753485 20510 tablet_service.cc:1467] Processing CreateTablet for tablet ba14db87c2724427b67c5beea7f1ce4a (DEFAULT_TABLE table=test-table [id=d3d190602f8e4443acb1d0732e6a51dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:13.754884 20510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba14db87c2724427b67c5beea7f1ce4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:13.756927 20379 tablet_service.cc:1467] Processing CreateTablet for tablet ba14db87c2724427b67c5beea7f1ce4a (DEFAULT_TABLE table=test-table [id=d3d190602f8e4443acb1d0732e6a51dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:13.756915 20640 tablet_service.cc:1467] Processing CreateTablet for tablet ba14db87c2724427b67c5beea7f1ce4a (DEFAULT_TABLE table=test-table [id=d3d190602f8e4443acb1d0732e6a51dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:13.758265 20379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba14db87c2724427b67c5beea7f1ce4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:13.758265 20640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba14db87c2724427b67c5beea7f1ce4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:13.773373 20723 tablet_bootstrap.cc:492] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: Bootstrap starting.
I20260927 22:56:13.777971 20723 tablet_bootstrap.cc:654] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:13.778352 20725 tablet_bootstrap.cc:492] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785: Bootstrap starting.
I20260927 22:56:13.779392 20723 log.cc:824] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:13.781189 20725 tablet_bootstrap.cc:654] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:13.782836 20723 tablet_bootstrap.cc:492] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: No bootstrap required, opened a new log
I20260927 22:56:13.782843 20724 tablet_bootstrap.cc:492] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: Bootstrap starting.
I20260927 22:56:13.783106 20723 ts_tablet_manager.cc:1397] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 22:56:13.785437 20724 tablet_bootstrap.cc:654] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:13.786126 20444 heartbeater.cc:503] Master 127.0.97.62:46329 was elected leader, sending a full tablet report...
I20260927 22:56:13.786561 20725 log.cc:824] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:13.786571 20724 log.cc:824] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:13.790541 20725 tablet_bootstrap.cc:492] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785: No bootstrap required, opened a new log
I20260927 22:56:13.790805 20725 ts_tablet_manager.cc:1397] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260927 22:56:13.792128 20723 raft_consensus.cc:348] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37291dc9f04acf86561fd70f3e7450" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46835 } } peers { permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 } } peers { permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 } }
I20260927 22:56:13.792549 20723 raft_consensus.cc:374] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:13.792644 20723 raft_consensus.cc:729] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c37291dc9f04acf86561fd70f3e7450, State: Initialized, Role: FOLLOWER
I20260927 22:56:13.793362 20724 tablet_bootstrap.cc:492] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: No bootstrap required, opened a new log
I20260927 22:56:13.793231 20723 consensus_queue.cc:260] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37291dc9f04acf86561fd70f3e7450" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46835 } } peers { permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 } } peers { permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 } }
I20260927 22:56:13.793709 20724 ts_tablet_manager.cc:1397] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260927 22:56:13.794359 20575 heartbeater.cc:503] Master 127.0.97.62:46329 was elected leader, sending a full tablet report...
I20260927 22:56:13.794945 20723 ts_tablet_manager.cc:1428] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260927 22:56:13.800160 20725 raft_consensus.cc:348] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37291dc9f04acf86561fd70f3e7450" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46835 } } peers { permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 } } peers { permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 } }
I20260927 22:56:13.800503 20725 raft_consensus.cc:374] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:13.800606 20725 raft_consensus.cc:729] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ca388559434c3fa58d4957d5509785, State: Initialized, Role: FOLLOWER
I20260927 22:56:13.801230 20725 consensus_queue.cc:260] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37291dc9f04acf86561fd70f3e7450" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46835 } } peers { permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 } } peers { permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 } }
I20260927 22:56:13.802057 20724 raft_consensus.cc:348] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37291dc9f04acf86561fd70f3e7450" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46835 } } peers { permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 } } peers { permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 } }
I20260927 22:56:13.802484 20724 raft_consensus.cc:374] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:13.802589 20724 raft_consensus.cc:729] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20169a683d314643832617d78f80960e, State: Initialized, Role: FOLLOWER
I20260927 22:56:13.802946 20725 ts_tablet_manager.cc:1428] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 22:56:13.803298 20724 consensus_queue.cc:260] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37291dc9f04acf86561fd70f3e7450" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46835 } } peers { permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 } } peers { permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 } }
I20260927 22:56:13.804359 20705 heartbeater.cc:503] Master 127.0.97.62:46329 was elected leader, sending a full tablet report...
I20260927 22:56:13.804975 20724 ts_tablet_manager.cc:1428] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 22:56:13.893791 20729 raft_consensus.cc:482] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:56:13.894124 20729 raft_consensus.cc:504] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37291dc9f04acf86561fd70f3e7450" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46835 } } peers { permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 } } peers { permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 } }
I20260927 22:56:13.895733 20729 leader_election.cc:288] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67ca388559434c3fa58d4957d5509785 (127.0.97.1:38905), 20169a683d314643832617d78f80960e (127.0.97.3:33223)
I20260927 22:56:13.902596 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.896083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51385 (local address 127.0.97.1:38905)
0927 22:56:13.896324 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:13.896332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:13.896741 (+ 409us) server_negotiation.cc:407] Connection header received
0927 22:56:13.896960 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:13.896968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:13.897071 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:13.897170 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:13.897823 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.898736 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.900680 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.901651 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.901758 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.901822 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:13.902215 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 22:56:13.902339 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:56:13.902643 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.896027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38905 (local address 127.0.97.2:51385)
0927 22:56:13.896625 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:13.896644 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:13.896816 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:13.897254 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:13.897266 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:13.897290 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:13.897665 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.897677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.898868 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.898873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.900500 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.900512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.901850 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.901857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.902054 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.902090 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:56:13.902179 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 22:56:13.902307 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":1,"thread_start_us":285,"threads_started":1}
I20260927 22:56:13.902998 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.896990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53861 (local address 127.0.97.3:33223)
0927 22:56:13.897213 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:13.897221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:13.897483 (+ 262us) server_negotiation.cc:407] Connection header received
0927 22:56:13.897617 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:13.897624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:13.897704 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:13.897795 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:13.898349 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.899231 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.901121 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.902057 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.902190 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.902280 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:13.902635 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 22:56:13.902766 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 22:56:13.903281 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.896775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33223 (local address 127.0.97.2:53861)
0927 22:56:13.897365 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:13.897386 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:13.897489 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:13.897823 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:13.897826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:13.897848 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:13.898209 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.898221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.899371 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.899375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.900942 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:13.900957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.902266 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.902272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:13.902447 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.902488 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:56:13.902598 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 22:56:13.902892 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":22,"thread_start_us":275,"threads_started":1}
I20260927 22:56:13.903648 20399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba14db87c2724427b67c5beea7f1ce4a" candidate_uuid: "1c37291dc9f04acf86561fd70f3e7450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67ca388559434c3fa58d4957d5509785" is_pre_election: true
I20260927 22:56:13.904224 20399 raft_consensus.cc:2454] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c37291dc9f04acf86561fd70f3e7450 in term 0.
I20260927 22:56:13.904266 20660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba14db87c2724427b67c5beea7f1ce4a" candidate_uuid: "1c37291dc9f04acf86561fd70f3e7450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20169a683d314643832617d78f80960e" is_pre_election: true
I20260927 22:56:13.904783 20660 raft_consensus.cc:2454] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c37291dc9f04acf86561fd70f3e7450 in term 0.
I20260927 22:56:13.904843 20464 leader_election.cc:302] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37291dc9f04acf86561fd70f3e7450, 67ca388559434c3fa58d4957d5509785; no voters:
I20260927 22:56:13.905179 20729 raft_consensus.cc:2781] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:56:13.905344 20729 raft_consensus.cc:482] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:56:13.905468 20729 raft_consensus.cc:3037] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:13.907634 20729 raft_consensus.cc:504] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37291dc9f04acf86561fd70f3e7450" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46835 } } peers { permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 } } peers { permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 } }
I20260927 22:56:13.908247 20729 leader_election.cc:288] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [CANDIDATE]: Term 1 election: Requested vote from peers 67ca388559434c3fa58d4957d5509785 (127.0.97.1:38905), 20169a683d314643832617d78f80960e (127.0.97.3:33223)
I20260927 22:56:13.908644 20399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba14db87c2724427b67c5beea7f1ce4a" candidate_uuid: "1c37291dc9f04acf86561fd70f3e7450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67ca388559434c3fa58d4957d5509785"
I20260927 22:56:13.908846 20399 raft_consensus.cc:3037] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:13.908722 20660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba14db87c2724427b67c5beea7f1ce4a" candidate_uuid: "1c37291dc9f04acf86561fd70f3e7450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20169a683d314643832617d78f80960e"
I20260927 22:56:13.909039 20660 raft_consensus.cc:3037] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:13.911127 20399 raft_consensus.cc:2454] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37291dc9f04acf86561fd70f3e7450 in term 1.
I20260927 22:56:13.911473 20464 leader_election.cc:302] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c37291dc9f04acf86561fd70f3e7450, 67ca388559434c3fa58d4957d5509785; no voters:
I20260927 22:56:13.911731 20729 raft_consensus.cc:2781] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:13.911801 20660 raft_consensus.cc:2454] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c37291dc9f04acf86561fd70f3e7450 in term 1.
I20260927 22:56:13.912204 20729 raft_consensus.cc:686] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 1 LEADER]: Becoming Leader. State: Replica: 1c37291dc9f04acf86561fd70f3e7450, State: Running, Role: LEADER
I20260927 22:56:13.912911 20729 consensus_queue.cc:237] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37291dc9f04acf86561fd70f3e7450" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46835 } } peers { permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 } } peers { permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 } }
I20260927 22:56:13.917366 20258 catalog_manager.cc:5495] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c37291dc9f04acf86561fd70f3e7450 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1c37291dc9f04acf86561fd70f3e7450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c37291dc9f04acf86561fd70f3e7450" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 } health_report { overall_health: UNKNOWN } } }
W20260927 22:56:13.926173 20706 tablet.cc:2369] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:13.939354 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.933805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.0.97.2:46835)
0927 22:56:13.934016 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:13.934020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:13.934143 (+ 123us) server_negotiation.cc:407] Connection header received
0927 22:56:13.934311 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:13.934315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:13.934487 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:13.934696 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:13.935044 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.935890 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.937399 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.937863 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.937958 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.938397 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:13.938872 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:13.938977 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:13.939103 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 22:56:13.939162 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:56:13.947103 20660 raft_consensus.cc:1261] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e [term 1 FOLLOWER]: Refusing update from remote peer 1c37291dc9f04acf86561fd70f3e7450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:13.947170 20399 raft_consensus.cc:1261] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785 [term 1 FOLLOWER]: Refusing update from remote peer 1c37291dc9f04acf86561fd70f3e7450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:13.948156 20734 consensus_queue.cc:1035] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20169a683d314643832617d78f80960e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:13.948494 20729 consensus_queue.cc:1035] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67ca388559434c3fa58d4957d5509785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:13.960114 20737 mvcc.cc:205] Tried to move back new op lower bound from 7334091874077261824 to 7334091873951383552. Current Snapshot: MvccSnapshot[applied={T|T < 7334091874077261824}]
I20260927 22:56:13.965675 20739 mvcc.cc:205] Tried to move back new op lower bound from 7334091874077261824 to 7334091873951383552. Current Snapshot: MvccSnapshot[applied={T|T < 7334091874077261824 or (T in {7334091874077261824})}]
I20260927 22:56:13.972585 20258 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:13.972847 20258 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:13.976516 20258 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72e9f6c43bb14aa0b8fb979973318ee2: Sending DeleteTablet for 3 replicas of tablet ba14db87c2724427b67c5beea7f1ce4a
I20260927 22:56:13.977483 20510 tablet_service.cc:1514] Processing DeleteTablet for tablet ba14db87c2724427b67c5beea7f1ce4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:13 UTC) from {username='slave'} at 127.0.0.1:40096
I20260927 22:56:13.977715 20379 tablet_service.cc:1514] Processing DeleteTablet for tablet ba14db87c2724427b67c5beea7f1ce4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:13 UTC) from {username='slave'} at 127.0.0.1:39984
I20260927 22:56:13.977754 20640 tablet_service.cc:1514] Processing DeleteTablet for tablet ba14db87c2724427b67c5beea7f1ce4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:13 UTC) from {username='slave'} at 127.0.0.1:54496
I20260927 22:56:13.978979 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 20317
I20260927 22:56:13.990716 20747 tablet_replica.cc:330] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785: stopping tablet replica
I20260927 22:56:13.991227 20747 raft_consensus.cc:2229] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:13.991885 20747 raft_consensus.cc:2258] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:13.994037 20747 ts_tablet_manager.cc:1905] T ba14db87c2724427b67c5beea7f1ce4a P 67ca388559434c3fa58d4957d5509785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:13.997689 20746 tablet_replica.cc:330] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: stopping tablet replica
I20260927 22:56:13.998119 20748 tablet_replica.cc:330] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: stopping tablet replica
I20260927 22:56:13.998157 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:13.980520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.0.97.1:38905)
0927 22:56:13.980683 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:13.980689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:13.990061 (+ 9372us) server_negotiation.cc:407] Connection header received
0927 22:56:13.990269 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:13.990277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:13.990431 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:13.990548 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:13.991149 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.992375 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.994332 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:13.994989 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:13.995084 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:13.996442 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:13.996453 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:13.996474 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:13.997186 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:13.997240 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:13.997262 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:13.997357 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:13.997681 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 22:56:13.997775 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:56:13.998131 20746 raft_consensus.cc:2229] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:56:13.998517 20748 raft_consensus.cc:2229] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:13.998781 20746 raft_consensus.cc:2258] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:13.998966 20748 raft_consensus.cc:2258] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:13.999223 20439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:14.001309 20748 ts_tablet_manager.cc:1905] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:14.003855 20746 ts_tablet_manager.cc:1905] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:14.009249 20746 ts_tablet_manager.cc:1918] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:14.009444 20746 log.cc:1196] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-1/wal/wals/ba14db87c2724427b67c5beea7f1ce4a
I20260927 22:56:14.009964 20746 ts_tablet_manager.cc:1939] T ba14db87c2724427b67c5beea7f1ce4a P 1c37291dc9f04acf86561fd70f3e7450: Deleting consensus metadata
I20260927 22:56:14.011627 20243 catalog_manager.cc:4841] TS 1c37291dc9f04acf86561fd70f3e7450 (127.0.97.2:46835): tablet ba14db87c2724427b67c5beea7f1ce4a (table test-table [id=d3d190602f8e4443acb1d0732e6a51dc]) successfully deleted
I20260927 22:56:14.014528 20748 ts_tablet_manager.cc:1918] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:14.014701 20748 log.cc:1196] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: Deleting WAL directory at /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549465302610-388-0/minicluster-data/ts-2/wal/wals/ba14db87c2724427b67c5beea7f1ce4a
I20260927 22:56:14.015152 20748 ts_tablet_manager.cc:1939] T ba14db87c2724427b67c5beea7f1ce4a P 20169a683d314643832617d78f80960e: Deleting consensus metadata
I20260927 22:56:14.016501 20243 catalog_manager.cc:4841] TS 20169a683d314643832617d78f80960e (127.0.97.3:33223): tablet ba14db87c2724427b67c5beea7f1ce4a (table test-table [id=d3d190602f8e4443acb1d0732e6a51dc]) successfully deleted
I20260927 22:56:14.116722 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 20317
I20260927 22:56:14.133733 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 20448
W20260927 22:56:14.133731 20243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38905 (error 108)
I20260927 22:56:14.135918 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:14.135448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38905 (local address 127.0.0.1:39992)
0927 22:56:14.135630 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:14.135750 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 22:56:14.136255 20243 catalog_manager.cc:4584] TS 67ca388559434c3fa58d4957d5509785 (127.0.97.1:38905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba14db87c2724427b67c5beea7f1ce4a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38905: connect: Connection refused (error 111)
I20260927 22:56:14.144456 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:14.134824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.0.97.2:46835)
0927 22:56:14.135016 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:14.135024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:14.135049 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:56:14.135316 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:14.135322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:14.135445 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:14.135564 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:14.136149 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:14.137044 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:14.138442 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:14.138869 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:14.139008 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:14.143104 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:14.143113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:14.143126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:14.143705 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:14.143751 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:14.143776 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:14.143891 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:14.144053 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 22:56:14.144145 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:56:14.145081 20570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:14.189929 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:14.189427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38905 (local address 127.0.0.1:39994)
0927 22:56:14.189689 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:14.189777 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 22:56:14.190881 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:14.190548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38905 (local address 127.0.0.1:40000)
0927 22:56:14.190692 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:14.190765 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:56:14.251386 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:14.250783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38905 (local address 127.0.0.1:40008)
0927 22:56:14.251018 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:14.251150 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 22:56:14.252827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 20448
I20260927 22:56:14.253139 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:14.252646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38905 (local address 127.0.0.1:40014)
0927 22:56:14.252858 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:14.252961 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 22:56:14.270900 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 20578
I20260927 22:56:14.277375 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:14.271665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.0.97.3:33223)
0927 22:56:14.271823 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:14.271828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:14.271845 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:56:14.271998 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:14.272002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:14.272087 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:14.272170 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:14.272567 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:14.273262 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:14.274618 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:14.275095 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:14.275191 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:14.276136 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:14.276144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:14.276158 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:14.276720 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:14.276766 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:14.276789 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:14.276872 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:14.276983 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 22:56:14.277064 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:56:14.278169 20700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:14.335265 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:14.334669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38905 (local address 127.0.0.1:40018)
0927 22:56:14.334916 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:14.335077 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 22:56:14.336651 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:14.336189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38905 (local address 127.0.0.1:40024)
0927 22:56:14.336426 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:14.336513 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 22:56:14.366075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 20578
I20260927 22:56:14.382201 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 20227
I20260927 22:56:14.383292 20288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:14.492743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 20227
2026-09-27T22:56:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2757 ms)
[----------] 6 tests from EncryptionPolicyTest (21212 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:14 dist-test-slave-df53 krb5kdc[20757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:14 dist-test-slave-df53 krb5kdc[20757](info): set up 2 sockets
Sep 27 22:56:14 dist-test-slave-df53 krb5kdc[20757](info): commencing operation
krb5kdc: starting...
W20260927 22:56:15.602370 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:15 dist-test-slave-df53 krb5kdc[20757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:15.833719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:16.149267 20770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:16.149628 20770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:16.149793 20770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:16.156982 20770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:16.157137 20770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:16.157240 20770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:16.157346 20770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:16.157434 20770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:16.167084 20770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:16.168316 20770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:16.169652 20770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:16.181180 20776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:16.181229 20778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:16.181226 20775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:16.181945 20770 server_base.cc:1034] running on GCE node
I20260927 22:56:16.182711 20770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:16.184094 20770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:16.185320 20770 hybrid_clock.cc:648] HybridClock initialized: now 1790549776185286 us; error 44 us; skew 500 ppm
I20260927 22:56:16.185724 20770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:16 dist-test-slave-df53 krb5kdc[20757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:16.188725 20770 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:16.190215 20770 webserver.cc:460] Webserver started at http://127.0.0.1:45683/ using document root <none> and password file <none>
I20260927 22:56:16.190864 20770 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:16.191025 20770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:16.191323 20770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:16.194284 20770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "8b16add10cc84c30acbf91ac61b2e49a"
format_stamp: "Formatted at 2026-09-27 22:56:16 on dist-test-slave-df53"
server_key: "6310bf2401d1746bf19572e1c2eb8a07"
server_key_iv: "d3364717696096198edc83d1e980f22f"
server_key_version: "encryptionkey@0"
I20260927 22:56:16.195029 20770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "8b16add10cc84c30acbf91ac61b2e49a"
format_stamp: "Formatted at 2026-09-27 22:56:16 on dist-test-slave-df53"
server_key: "6310bf2401d1746bf19572e1c2eb8a07"
server_key_iv: "d3364717696096198edc83d1e980f22f"
server_key_version: "encryptionkey@0"
I20260927 22:56:16.199977 20770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:56:16.203274 20785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:16.204578 20770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:56:16.204754 20770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "8b16add10cc84c30acbf91ac61b2e49a"
format_stamp: "Formatted at 2026-09-27 22:56:16 on dist-test-slave-df53"
server_key: "6310bf2401d1746bf19572e1c2eb8a07"
server_key_iv: "d3364717696096198edc83d1e980f22f"
server_key_version: "encryptionkey@0"
I20260927 22:56:16.204950 20770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:16.224709 20770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:16.228813 20770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:16.229096 20770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:16.248551 20770 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40193
I20260927 22:56:16.248625 20836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40193 every 8 connection(s)
I20260927 22:56:16.249948 20770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:16.253479 20837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:16.255146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 20770
I20260927 22:56:16.255376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:16.256006 388 external_mini_cluster.cc:1428] Setting key 493a950e2bfb5e41dbbf58cbe8c1a02d
I20260927 22:56:16.267724 20837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a: Bootstrap starting.
Sep 27 22:56:16 dist-test-slave-df53 krb5kdc[20757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:56:16.270624 20837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:16.271960 20837 log.cc:824] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:16.275368 20837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a: No bootstrap required, opened a new log
I20260927 22:56:16.279589 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:16.257915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50575 (local address 127.0.0.1:40193)
0927 22:56:16.258736 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:16.258756 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:16.258810 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:56:16.260494 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:16.260525 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:16.261024 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:16.261455 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:16.262760 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:16.264792 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:16.266103 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:16.267100 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:16.267280 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:16.271133 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:16.271143 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:16.271162 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:16.271202 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:16.275118 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:16.275958 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:16.275968 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:16.275975 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:16.276183 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:16.276592 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:16.276597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:16.276600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:16.277263 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:16.277622 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:16.277982 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 22:56:16.278304 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":346,"threads_started":1}
I20260927 22:56:16.283478 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:16.285022 20837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b16add10cc84c30acbf91ac61b2e49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } }
I20260927 22:56:16.285388 20837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:16.285518 20837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b16add10cc84c30acbf91ac61b2e49a, State: Initialized, Role: FOLLOWER
I20260927 22:56:16.286080 20837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b16add10cc84c30acbf91ac61b2e49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } }
I20260927 22:56:16.286321 20837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:16.286513 20837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:16.286728 20837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:16.289333 20837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b16add10cc84c30acbf91ac61b2e49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } }
I20260927 22:56:16.289855 20837 leader_election.cc:302] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b16add10cc84c30acbf91ac61b2e49a; no voters:
I20260927 22:56:16.290496 20837 leader_election.cc:288] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:16.290599 20847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:16.292963 20837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:16.297639 20847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [term 1 LEADER]: Becoming Leader. State: Replica: 8b16add10cc84c30acbf91ac61b2e49a, State: Running, Role: LEADER
I20260927 22:56:16.298439 20847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b16add10cc84c30acbf91ac61b2e49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } }
I20260927 22:56:16.301493 20848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b16add10cc84c30acbf91ac61b2e49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b16add10cc84c30acbf91ac61b2e49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } } }
I20260927 22:56:16.301944 20848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:16.302953 20856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:16.304600 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:16.288075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60125 (local address 127.0.0.1:40193)
0927 22:56:16.288233 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:16.288256 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:16.288661 (+ 405us) server_negotiation.cc:407] Connection header received
0927 22:56:16.288850 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:16.288855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:16.288923 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:16.289019 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:16.290552 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:16.291761 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:16.293184 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:16.293723 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:16.293853 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:16.296474 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:16.296479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:16.296482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:16.296517 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:16.299840 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:16.300733 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:16.300741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:16.300747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:16.300930 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:16.301496 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:16.301504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:16.301510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:16.302055 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:16.302378 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:16.303828 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0927 22:56:16.304077 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:56:16.304657 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b16add10cc84c30acbf91ac61b2e49a. Latest consensus state: current_term: 1 leader_uuid: "8b16add10cc84c30acbf91ac61b2e49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b16add10cc84c30acbf91ac61b2e49a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40193 } } }
I20260927 22:56:16.304953 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:16.315446 20856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:16.326195 20856 catalog_manager.cc:1347] Generated new cluster ID: 7ced24fb8cc44594a880e239b05e6c29
I20260927 22:56:16.326321 20856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:16.335095 20856 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:16.336562 20856 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:16.343205 20856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b16add10cc84c30acbf91ac61b2e49a: Generated new TSK 0
I20260927 22:56:16.344130 20856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:16.354736 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 20770
I20260927 22:56:16.356720 20832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:16.448153 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 20770
W20260927 22:56:16.464886 20755 connection.cc:535] client connection to 127.0.0.1:40193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40193): Connection reset by peer (error 104)
2026-09-27T22:56:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:16 dist-test-slave-df53 krb5kdc[20869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:16 dist-test-slave-df53 krb5kdc[20869](info): set up 2 sockets
Sep 27 22:56:16 dist-test-slave-df53 krb5kdc[20869](info): commencing operation
krb5kdc: starting...
W20260927 22:56:17.541896 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:17 dist-test-slave-df53 krb5kdc[20869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:17.780390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:18.101866 20882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:18.102126 20882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:18.102283 20882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:18.109313 20882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:18.109412 20882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:18.109500 20882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:18.109624 20882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:18.109687 20882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:18.119285 20882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36929
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:18.120517 20882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:18.121848 20882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:18.133174 20887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:18.133237 20888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:18.133419 20890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:18.134207 20882 server_base.cc:1034] running on GCE node
I20260927 22:56:18.134881 20882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:18.136142 20882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:18.137341 20882 hybrid_clock.cc:648] HybridClock initialized: now 1790549778137307 us; error 31 us; skew 500 ppm
I20260927 22:56:18.137694 20882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:18 dist-test-slave-df53 krb5kdc[20869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:18.141384 20882 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:18.142854 20882 webserver.cc:460] Webserver started at http://127.0.0.1:39323/ using document root <none> and password file <none>
I20260927 22:56:18.143527 20882 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:18.143657 20882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:18.143973 20882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:18.147035 20882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-df53"
server_key: "d87d0e58761571cb2b3e62707f505f79"
server_key_iv: "319b85b3c812f7fa551395c61681a9eb"
server_key_version: "encryptionkey@0"
I20260927 22:56:18.147764 20882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-df53"
server_key: "d87d0e58761571cb2b3e62707f505f79"
server_key_iv: "319b85b3c812f7fa551395c61681a9eb"
server_key_version: "encryptionkey@0"
I20260927 22:56:18.152666 20882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:56:18.155893 20897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:18.157083 20882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 22:56:18.157254 20882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-df53"
server_key: "d87d0e58761571cb2b3e62707f505f79"
server_key_iv: "319b85b3c812f7fa551395c61681a9eb"
server_key_version: "encryptionkey@0"
I20260927 22:56:18.157434 20882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:18.192752 20882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:18.196748 20882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:18.197022 20882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:18.215358 20882 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36929
I20260927 22:56:18.215387 20948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36929 every 8 connection(s)
I20260927 22:56:18.216689 20882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:18.220258 20949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:18.222555 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 20882
I20260927 22:56:18.222728 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:18.223225 388 external_mini_cluster.cc:1428] Setting key f25724725c3f5be10114485a557a7553
I20260927 22:56:18.232200 20949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1: Bootstrap starting.
I20260927 22:56:18.235656 20949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:18.236407 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:18.224922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50353 (local address 127.0.0.1:36929)
0927 22:56:18.225767 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:18.225778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:18.225819 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:56:18.227481 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:18.227502 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:18.227901 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:18.228196 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:18.228746 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:18.230156 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:18.232019 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:18.232712 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:18.232831 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:18.234170 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:18.234177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:18.234189 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:18.234724 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:18.234794 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:18.234817 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:18.234936 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:18.235242 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 22:56:18.235412 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":340,"threads_started":1}
I20260927 22:56:18.236725 20949 log.cc:824] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:18.239773 20949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1: No bootstrap required, opened a new log
I20260927 22:56:18.240693 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:18.250275 20949 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } }
I20260927 22:56:18.250741 20949 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:18.250849 20949 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e4ca08a64b4d78a1ff3792e6b8a8a1, State: Initialized, Role: FOLLOWER
I20260927 22:56:18.251520 20949 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } }
I20260927 22:56:18.251813 20949 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:18.251961 20949 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:18.252146 20949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:18.252451 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:18.245839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43423 (local address 127.0.0.1:36929)
0927 22:56:18.245981 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:18.245985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:18.246307 (+ 322us) server_negotiation.cc:407] Connection header received
0927 22:56:18.246478 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:18.246481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:18.246547 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:18.246626 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:18.247188 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:18.247922 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:18.249877 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:18.250285 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:18.250373 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:18.251663 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:18.251668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:18.251671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:18.251686 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:18.251725 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:18.251763 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:18.251883 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:18.252096 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 22:56:18.252153 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 22:56:18.254966 20949 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } }
I20260927 22:56:18.255446 20949 leader_election.cc:302] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1e4ca08a64b4d78a1ff3792e6b8a8a1; no voters:
I20260927 22:56:18.255970 20949 leader_election.cc:288] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:18.256096 20959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:18.257156 20959 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [term 1 LEADER]: Becoming Leader. State: Replica: d1e4ca08a64b4d78a1ff3792e6b8a8a1, State: Running, Role: LEADER
I20260927 22:56:18.257709 20959 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } }
I20260927 22:56:18.258062 20949 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:18.260425 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } } }
I20260927 22:56:18.260783 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:18.261222 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1e4ca08a64b4d78a1ff3792e6b8a8a1. Latest consensus state: current_term: 1 leader_uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1e4ca08a64b4d78a1ff3792e6b8a8a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36929 } } }
I20260927 22:56:18.261631 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:18.266832 20966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:18.272205 20966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:18.280885 20966 catalog_manager.cc:1347] Generated new cluster ID: f644af929395406a8d9c7316f5a37358
I20260927 22:56:18.281006 20966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:18.292954 20966 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:18.294608 20966 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:18.302412 20966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1e4ca08a64b4d78a1ff3792e6b8a8a1: Generated new TSK 0
I20260927 22:56:18.303217 20966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:18.308173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 20882
W20260927 22:56:18.309249 20944 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50353
W20260927 22:56:18.310007 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 20882: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:18.310179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 20882
W20260927 22:56:18.326218 20867 connection.cc:535] client connection to 127.0.0.1:36929 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36929): Connection reset by peer (error 104)
2026-09-27T22:56:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1860 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:18 dist-test-slave-df53 krb5kdc[20980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:18 dist-test-slave-df53 krb5kdc[20980](info): set up 2 sockets
Sep 27 22:56:18 dist-test-slave-df53 krb5kdc[20980](info): commencing operation
krb5kdc: starting...
W20260927 22:56:19.425997 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:19 dist-test-slave-df53 krb5kdc[20980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:19.664705 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:19.981119 20993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:19.981398 20993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:19.981532 20993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:19.988628 20993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:19.988730 20993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:19.988790 20993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:19.988852 20993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:19.988895 20993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:19.998544 20993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:19.999675 20993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:20.000859 20993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:20.012625 20998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:20.012696 21001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:20.012662 20999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:20.013384 20993 server_base.cc:1034] running on GCE node
I20260927 22:56:20.014205 20993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:20.015461 20993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:20.016659 20993 hybrid_clock.cc:648] HybridClock initialized: now 1790549780016615 us; error 41 us; skew 500 ppm
I20260927 22:56:20.016989 20993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:20 dist-test-slave-df53 krb5kdc[20980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:20.026484 20993 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:20.027946 20993 webserver.cc:460] Webserver started at http://127.0.0.1:40251/ using document root <none> and password file <none>
I20260927 22:56:20.028579 20993 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:20.028676 20993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:20.028920 20993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:20.031802 20993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "7d16a18f48ab484fab4f26e1ad230847"
format_stamp: "Formatted at 2026-09-27 22:56:20 on dist-test-slave-df53"
server_key: "242004816546febc1448030d245511e2"
server_key_iv: "1da44f574177c5553e2a74a4280fc1ae"
server_key_version: "encryptionkey@0"
I20260927 22:56:20.032480 20993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "7d16a18f48ab484fab4f26e1ad230847"
format_stamp: "Formatted at 2026-09-27 22:56:20 on dist-test-slave-df53"
server_key: "242004816546febc1448030d245511e2"
server_key_iv: "1da44f574177c5553e2a74a4280fc1ae"
server_key_version: "encryptionkey@0"
I20260927 22:56:20.037305 20993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:56:20.040542 21008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:20.041862 20993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:56:20.042026 20993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "7d16a18f48ab484fab4f26e1ad230847"
format_stamp: "Formatted at 2026-09-27 22:56:20 on dist-test-slave-df53"
server_key: "242004816546febc1448030d245511e2"
server_key_iv: "1da44f574177c5553e2a74a4280fc1ae"
server_key_version: "encryptionkey@0"
I20260927 22:56:20.042210 20993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:20.047789 20993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:20.051550 20993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:20.051817 20993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:20.070030 20993 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32823
I20260927 22:56:20.070116 21059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32823 every 8 connection(s)
I20260927 22:56:20.071364 20993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:20.074925 21060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:20.076083 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 20993
I20260927 22:56:20.076261 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:20.076686 388 external_mini_cluster.cc:1428] Setting key 0e0a2eab4f6cd4963e6229270e7f3bc8
I20260927 22:56:20.084923 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:20.078447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43519 (local address 127.0.0.1:32823)
0927 22:56:20.079243 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:20.079258 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:20.079305 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:56:20.081241 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:20.081269 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:20.081745 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:20.082185 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:20.083099 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:20.083107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:20.083124 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:20.083139 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:20.083199 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:20.083227 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:20.083428 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:20.083588 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 22:56:20.083758 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":306,"threads_started":1}
I20260927 22:56:20.088373 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:20.088593 21060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847: Bootstrap starting.
I20260927 22:56:20.092365 21060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:20.093688 21060 log.cc:824] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:20.096040 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:20.093412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54685 (local address 127.0.0.1:32823)
0927 22:56:20.093656 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:20.093665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:20.094051 (+ 386us) server_negotiation.cc:407] Connection header received
0927 22:56:20.094254 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:20.094259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:20.094330 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:20.094446 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:20.095352 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:20.095356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:20.095359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:20.095372 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:20.095406 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:20.095420 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:20.095510 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:20.095704 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 22:56:20.095748 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:56:20.096743 21060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847: No bootstrap required, opened a new log
I20260927 22:56:20.103652 21060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a18f48ab484fab4f26e1ad230847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32823 } }
I20260927 22:56:20.104010 21060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:20.104115 21060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d16a18f48ab484fab4f26e1ad230847, State: Initialized, Role: FOLLOWER
I20260927 22:56:20.104616 21060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a18f48ab484fab4f26e1ad230847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32823 } }
I20260927 22:56:20.104848 21060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:20.104980 21060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:20.105144 21060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:20.107347 21060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a18f48ab484fab4f26e1ad230847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32823 } }
I20260927 22:56:20.107805 21060 leader_election.cc:302] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d16a18f48ab484fab4f26e1ad230847; no voters:
I20260927 22:56:20.108354 21060 leader_election.cc:288] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:20.108507 21070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:20.109165 21070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [term 1 LEADER]: Becoming Leader. State: Replica: 7d16a18f48ab484fab4f26e1ad230847, State: Running, Role: LEADER
I20260927 22:56:20.109704 21070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a18f48ab484fab4f26e1ad230847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32823 } }
I20260927 22:56:20.110394 21060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:20.112269 21071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d16a18f48ab484fab4f26e1ad230847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a18f48ab484fab4f26e1ad230847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32823 } } }
I20260927 22:56:20.112697 21071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:20.112298 21072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d16a18f48ab484fab4f26e1ad230847. Latest consensus state: current_term: 1 leader_uuid: "7d16a18f48ab484fab4f26e1ad230847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a18f48ab484fab4f26e1ad230847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32823 } } }
I20260927 22:56:20.114013 21072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:20.115329 21075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:20.120164 21075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:20.129185 21075 catalog_manager.cc:1347] Generated new cluster ID: 884dd7db6575403183df5ef4a9315287
I20260927 22:56:20.129302 21075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:20.152902 21075 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:20.153296 21075 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:20.164628 21075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d16a18f48ab484fab4f26e1ad230847: Generated new TSK 0
I20260927 22:56:20.165561 21075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:20.176465 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 20993
W20260927 22:56:20.177672 21055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43519
W20260927 22:56:20.178401 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 20993: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:20.178556 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 20993
2026-09-27T22:56:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:20 dist-test-slave-df53 krb5kdc[21091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:20 dist-test-slave-df53 krb5kdc[21091](info): set up 2 sockets
Sep 27 22:56:20 dist-test-slave-df53 krb5kdc[21091](info): commencing operation
krb5kdc: starting...
W20260927 22:56:21.265820 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:21 dist-test-slave-df53 krb5kdc[21091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:21.496296 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:21.812788 21104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:21.813130 21104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:21.813295 21104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:21.820595 21104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:21.820699 21104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:21.820806 21104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:21.820900 21104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:21.820983 21104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:21.830742 21104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:21.831966 21104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:21.833236 21104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:21.844586 21110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:21.844558 21112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:21.845098 21109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:21.845723 21104 server_base.cc:1034] running on GCE node
I20260927 22:56:21.846414 21104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:21.847739 21104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:21.848959 21104 hybrid_clock.cc:648] HybridClock initialized: now 1790549781848905 us; error 43 us; skew 500 ppm
I20260927 22:56:21.849289 21104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:21 dist-test-slave-df53 krb5kdc[21091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:21.852219 21104 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:21.853760 21104 webserver.cc:460] Webserver started at http://127.0.0.1:37267/ using document root <none> and password file <none>
I20260927 22:56:21.854398 21104 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:21.854540 21104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:21.854827 21104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:21.857626 21104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "e156f6fe14264edca45bc6fbfe7148ba"
format_stamp: "Formatted at 2026-09-27 22:56:21 on dist-test-slave-df53"
server_key: "8c908910fcc23794ef26303360ba6484"
server_key_iv: "7998fa025ac63c6cb5b9a91250818171"
server_key_version: "encryptionkey@0"
I20260927 22:56:21.858376 21104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "e156f6fe14264edca45bc6fbfe7148ba"
format_stamp: "Formatted at 2026-09-27 22:56:21 on dist-test-slave-df53"
server_key: "8c908910fcc23794ef26303360ba6484"
server_key_iv: "7998fa025ac63c6cb5b9a91250818171"
server_key_version: "encryptionkey@0"
I20260927 22:56:21.863113 21104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:56:21.866367 21119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:21.867617 21104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:21.867813 21104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "e156f6fe14264edca45bc6fbfe7148ba"
format_stamp: "Formatted at 2026-09-27 22:56:21 on dist-test-slave-df53"
server_key: "8c908910fcc23794ef26303360ba6484"
server_key_iv: "7998fa025ac63c6cb5b9a91250818171"
server_key_version: "encryptionkey@0"
I20260927 22:56:21.868036 21104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:21.881983 21104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:21.885638 21104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:21.885906 21104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:21.903470 21104 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42535
I20260927 22:56:21.903504 21170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42535 every 8 connection(s)
I20260927 22:56:21.904749 21104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:21.906162 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 21104
I20260927 22:56:21.906354 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:21.906883 388 external_mini_cluster.cc:1428] Setting key a6baa33ad6e81dbec50c1a194a904eae
I20260927 22:56:21.909237 21171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:56:21 dist-test-slave-df53 krb5kdc[21091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:56:21.921963 21171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba: Bootstrap starting.
I20260927 22:56:21.925149 21171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:21.926172 21171 log.cc:824] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:21.928777 21171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba: No bootstrap required, opened a new log
I20260927 22:56:21.928910 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:21.908573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54821 (local address 127.0.0.1:42535)
0927 22:56:21.909345 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:21.909361 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:21.909407 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:56:21.911033 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:21.911052 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:21.911419 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:21.911752 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:21.913463 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:21.914827 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:21.916616 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:21.917225 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:21.917350 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:21.922096 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:21.922107 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:21.922126 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:21.922165 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:21.924949 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:21.925727 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:21.925734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:21.925739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:21.925891 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:21.926272 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:21.926278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:21.926281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:21.926821 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:21.927133 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:21.927566 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 22:56:21.927973 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":365,"threads_started":1}
I20260927 22:56:21.932246 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:21.936479 21171 raft_consensus.cc:348] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e156f6fe14264edca45bc6fbfe7148ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260927 22:56:21.936852 21171 raft_consensus.cc:374] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:21.936950 21171 raft_consensus.cc:729] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e156f6fe14264edca45bc6fbfe7148ba, State: Initialized, Role: FOLLOWER
I20260927 22:56:21.937525 21171 consensus_queue.cc:260] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e156f6fe14264edca45bc6fbfe7148ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260927 22:56:21.937803 21171 raft_consensus.cc:388] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:21.937939 21171 raft_consensus.cc:482] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:21.938107 21171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:21.940176 21171 raft_consensus.cc:504] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e156f6fe14264edca45bc6fbfe7148ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260927 22:56:21.940649 21171 leader_election.cc:302] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e156f6fe14264edca45bc6fbfe7148ba; no voters:
I20260927 22:56:21.941269 21171 leader_election.cc:288] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:21.941392 21181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:21.943646 21171 sys_catalog.cc:564] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:21.948654 21181 raft_consensus.cc:686] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [term 1 LEADER]: Becoming Leader. State: Replica: e156f6fe14264edca45bc6fbfe7148ba, State: Running, Role: LEADER
I20260927 22:56:21.949365 21181 consensus_queue.cc:237] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e156f6fe14264edca45bc6fbfe7148ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } }
I20260927 22:56:21.950155 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:21.936332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50511 (local address 127.0.0.1:42535)
0927 22:56:21.936513 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:21.936519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:21.936862 (+ 343us) server_negotiation.cc:407] Connection header received
0927 22:56:21.937058 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:21.937063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:21.937149 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:21.937243 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:21.938841 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:21.939562 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:21.940947 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:21.941524 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:21.941745 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:21.944192 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:21.944198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:21.944203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:21.944242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:21.947491 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:21.948133 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:21.948138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:21.948142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:21.948314 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:21.948688 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:21.948693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:21.948696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:21.949038 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:21.949324 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:21.949576 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 22:56:21.949789 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:56:21.952718 21182 sys_catalog.cc:455] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e156f6fe14264edca45bc6fbfe7148ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e156f6fe14264edca45bc6fbfe7148ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } } }
I20260927 22:56:21.953080 21182 sys_catalog.cc:458] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:21.952653 21183 sys_catalog.cc:455] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader e156f6fe14264edca45bc6fbfe7148ba. Latest consensus state: current_term: 1 leader_uuid: "e156f6fe14264edca45bc6fbfe7148ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e156f6fe14264edca45bc6fbfe7148ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42535 } } }
I20260927 22:56:21.954447 21183 sys_catalog.cc:458] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:21.959233 21191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:21.963191 21191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:21.971482 21191 catalog_manager.cc:1347] Generated new cluster ID: 92704f6e0abf4857935411ba56219e03
I20260927 22:56:21.971581 21191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:21.988130 21191 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:21.989307 21191 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:21.995850 21191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e156f6fe14264edca45bc6fbfe7148ba: Generated new TSK 0
I20260927 22:56:21.996562 21191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:22.002660 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21104
I20260927 22:56:22.003916 21166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:22.091900 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 21104
2026-09-27T22:56:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:22 dist-test-slave-df53 krb5kdc[21203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:22 dist-test-slave-df53 krb5kdc[21203](info): set up 2 sockets
Sep 27 22:56:22 dist-test-slave-df53 krb5kdc[21203](info): commencing operation
krb5kdc: starting...
W20260927 22:56:23.177850 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:23 dist-test-slave-df53 krb5kdc[21203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:23.410413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:23.749745 21216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:23.750102 21216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:23.750252 21216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:23.757617 21216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:23.757733 21216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:23.757830 21216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:23.757927 21216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:23.757993 21216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:23.767843 21216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:23.769104 21216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:23.770514 21216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:23.781912 21221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:23.781973 21222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:23.782008 21224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:23.782754 21216 server_base.cc:1034] running on GCE node
I20260927 22:56:23.783556 21216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:23.784880 21216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:23.786095 21216 hybrid_clock.cc:648] HybridClock initialized: now 1790549783786054 us; error 36 us; skew 500 ppm
I20260927 22:56:23.786419 21216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:23 dist-test-slave-df53 krb5kdc[21203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:23.789410 21216 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:23.790838 21216 webserver.cc:460] Webserver started at http://127.0.0.1:33263/ using document root <none> and password file <none>
I20260927 22:56:23.791507 21216 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:23.791688 21216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:23.791989 21216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:23.794869 21216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "71e639f3325545a1a43827a874a65e21"
format_stamp: "Formatted at 2026-09-27 22:56:23 on dist-test-slave-df53"
server_key: "ceab78431f97dafc5e15930f30786e9d"
server_key_iv: "dc3c68e0c5a2ecb498ad85df146a0aea"
server_key_version: "encryptionkey@0"
I20260927 22:56:23.795603 21216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "71e639f3325545a1a43827a874a65e21"
format_stamp: "Formatted at 2026-09-27 22:56:23 on dist-test-slave-df53"
server_key: "ceab78431f97dafc5e15930f30786e9d"
server_key_iv: "dc3c68e0c5a2ecb498ad85df146a0aea"
server_key_version: "encryptionkey@0"
I20260927 22:56:23.800446 21216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 22:56:23.803681 21231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:23.804973 21216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:56:23.805140 21216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "71e639f3325545a1a43827a874a65e21"
format_stamp: "Formatted at 2026-09-27 22:56:23 on dist-test-slave-df53"
server_key: "ceab78431f97dafc5e15930f30786e9d"
server_key_iv: "dc3c68e0c5a2ecb498ad85df146a0aea"
server_key_version: "encryptionkey@0"
I20260927 22:56:23.805330 21216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:23.831722 21216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:23.835703 21216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:23.835983 21216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:23.853497 21216 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36921
I20260927 22:56:23.853533 21282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36921 every 8 connection(s)
I20260927 22:56:23.854849 21216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:23.858592 21283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:23.861874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 21216
I20260927 22:56:23.862087 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:23.862632 388 external_mini_cluster.cc:1428] Setting key e481526935bdf0d6743fb9251a5244b7
I20260927 22:56:23.870705 21283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21: Bootstrap starting.
I20260927 22:56:23.874293 21283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:23.875335 21283 log.cc:824] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:23.875684 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.864404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36053 (local address 127.0.0.1:36921)
0927 22:56:23.865143 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:23.865154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:23.865194 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:56:23.866866 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:23.866888 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:23.867281 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:23.867599 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:23.868150 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.869427 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.871345 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.871990 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.872125 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:23.873357 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:23.873364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:23.873377 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:23.873962 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:23.874035 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:23.874060 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:23.874142 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:23.874425 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 22:56:23.874733 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":357,"threads_started":1}
I20260927 22:56:23.878585 21283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21: No bootstrap required, opened a new log
I20260927 22:56:23.879487 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:23.888622 21283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e639f3325545a1a43827a874a65e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 22:56:23.889109 21283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:23.889220 21283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e639f3325545a1a43827a874a65e21, State: Initialized, Role: FOLLOWER
I20260927 22:56:23.889899 21283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e639f3325545a1a43827a874a65e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 22:56:23.890182 21283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:23.890338 21283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:23.890507 21283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:23.890983 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.884521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36919 (local address 127.0.0.1:36921)
0927 22:56:23.884691 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:23.884697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:23.885166 (+ 469us) server_negotiation.cc:407] Connection header received
0927 22:56:23.885294 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:23.885299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:23.885377 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:23.885472 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:23.886003 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.887090 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.888365 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.889033 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.889168 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:23.890268 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:23.890277 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:23.890282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:23.890301 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:23.890347 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:23.890362 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:23.890460 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:23.890615 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 22:56:23.890698 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:56:23.893291 21283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e639f3325545a1a43827a874a65e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 22:56:23.893821 21283 leader_election.cc:302] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71e639f3325545a1a43827a874a65e21; no voters:
I20260927 22:56:23.894343 21283 leader_election.cc:288] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:23.894467 21293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:23.895984 21293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [term 1 LEADER]: Becoming Leader. State: Replica: 71e639f3325545a1a43827a874a65e21, State: Running, Role: LEADER
I20260927 22:56:23.896301 21283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:23.896551 21293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e639f3325545a1a43827a874a65e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 22:56:23.899470 21295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71e639f3325545a1a43827a874a65e21. Latest consensus state: current_term: 1 leader_uuid: "71e639f3325545a1a43827a874a65e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e639f3325545a1a43827a874a65e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } } }
I20260927 22:56:23.899828 21295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:23.900177 21294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71e639f3325545a1a43827a874a65e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e639f3325545a1a43827a874a65e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } } }
I20260927 22:56:23.900537 21294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:23.902869 21301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:23.908100 21301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:23.916234 21301 catalog_manager.cc:1347] Generated new cluster ID: b3ab3ed59b2e477b966842c6d336303a
I20260927 22:56:23.916338 21301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:23.943269 21301 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:23.944660 21301 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:23.950891 21301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71e639f3325545a1a43827a874a65e21: Generated new TSK 0
I20260927 22:56:23.951730 21301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:23.966682 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21216
W20260927 22:56:23.967641 21278 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36053
W20260927 22:56:23.968276 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21216: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:23.968403 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 21216
2026-09-27T22:56:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:24 dist-test-slave-df53 krb5kdc[21314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:24 dist-test-slave-df53 krb5kdc[21314](info): set up 2 sockets
Sep 27 22:56:24 dist-test-slave-df53 krb5kdc[21314](info): commencing operation
krb5kdc: starting...
W20260927 22:56:25.065893 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:25 dist-test-slave-df53 krb5kdc[21314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:25.301923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:25.626112 21327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:25.626366 21327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:25.626535 21327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:25.633445 21327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:25.633544 21327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:25.633661 21327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:25.633760 21327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:25.633831 21327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:25.643153 21327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:25.644301 21327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:25.645552 21327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:25.657002 21333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:25.657078 21332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:25.657028 21335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:25.657828 21327 server_base.cc:1034] running on GCE node
I20260927 22:56:25.658526 21327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:25.659734 21327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:25.660933 21327 hybrid_clock.cc:648] HybridClock initialized: now 1790549785660889 us; error 44 us; skew 500 ppm
I20260927 22:56:25.661265 21327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:25 dist-test-slave-df53 krb5kdc[21314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:25.668529 21327 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:25.669914 21327 webserver.cc:460] Webserver started at http://127.0.0.1:40109/ using document root <none> and password file <none>
I20260927 22:56:25.670529 21327 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:25.670625 21327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:25.670871 21327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:25.673826 21327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "8d9f5d1ae1fc4a77895323c1596bc713"
format_stamp: "Formatted at 2026-09-27 22:56:25 on dist-test-slave-df53"
server_key: "688ec22a2d8f9684f216f62255939104"
server_key_iv: "028e87487d8bffda17c63fd40d1b8975"
server_key_version: "encryptionkey@0"
I20260927 22:56:25.674535 21327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "8d9f5d1ae1fc4a77895323c1596bc713"
format_stamp: "Formatted at 2026-09-27 22:56:25 on dist-test-slave-df53"
server_key: "688ec22a2d8f9684f216f62255939104"
server_key_iv: "028e87487d8bffda17c63fd40d1b8975"
server_key_version: "encryptionkey@0"
I20260927 22:56:25.679406 21327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:56:25.682902 21342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:25.684191 21327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:56:25.684368 21327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "8d9f5d1ae1fc4a77895323c1596bc713"
format_stamp: "Formatted at 2026-09-27 22:56:25 on dist-test-slave-df53"
server_key: "688ec22a2d8f9684f216f62255939104"
server_key_iv: "028e87487d8bffda17c63fd40d1b8975"
server_key_version: "encryptionkey@0"
I20260927 22:56:25.684561 21327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:25.689986 21327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:25.693814 21327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:25.694078 21327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:25.711930 21327 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36513
I20260927 22:56:25.711992 21393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36513 every 8 connection(s)
I20260927 22:56:25.713233 21327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:25.716650 21394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:25.722581 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 21327
I20260927 22:56:25.722781 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:25.723304 388 external_mini_cluster.cc:1428] Setting key 42a4e80007a5bcaed83cdc087fb9bb2e
I20260927 22:56:25.727203 21394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713: Bootstrap starting.
I20260927 22:56:25.730574 21394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:25.730664 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:25.724993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54159 (local address 127.0.0.1:36513)
0927 22:56:25.725804 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:25.725815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:25.725850 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:56:25.727558 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:25.727578 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:25.727874 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:25.728170 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:25.729190 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:25.729195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:25.729207 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:25.729220 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:25.729270 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:25.729290 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:25.729453 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:25.729581 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 22:56:25.729806 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":360,"threads_started":1}
I20260927 22:56:25.731611 21394 log.cc:824] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:25.734194 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:25.734488 21394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713: No bootstrap required, opened a new log
I20260927 22:56:25.740564 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:25.738255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56937 (local address 127.0.0.1:36513)
0927 22:56:25.738436 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:25.738446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:25.738833 (+ 387us) server_negotiation.cc:407] Connection header received
0927 22:56:25.738951 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:25.738957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:25.739013 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:25.739123 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:25.739921 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:25.739926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:25.739931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:25.739948 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:25.739988 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:25.740005 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:25.740112 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:25.740189 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 22:56:25.740246 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:56:25.742414 21394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9f5d1ae1fc4a77895323c1596bc713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260927 22:56:25.742722 21394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:25.742791 21394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d9f5d1ae1fc4a77895323c1596bc713, State: Initialized, Role: FOLLOWER
I20260927 22:56:25.743244 21394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9f5d1ae1fc4a77895323c1596bc713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260927 22:56:25.743438 21394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:25.743528 21394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:25.743654 21394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:25.745736 21394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9f5d1ae1fc4a77895323c1596bc713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260927 22:56:25.746220 21394 leader_election.cc:302] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d9f5d1ae1fc4a77895323c1596bc713; no voters:
I20260927 22:56:25.746898 21394 leader_election.cc:288] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:25.747008 21404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:25.747766 21404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [term 1 LEADER]: Becoming Leader. State: Replica: 8d9f5d1ae1fc4a77895323c1596bc713, State: Running, Role: LEADER
I20260927 22:56:25.748272 21404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9f5d1ae1fc4a77895323c1596bc713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260927 22:56:25.748901 21394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:25.751276 21405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d9f5d1ae1fc4a77895323c1596bc713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9f5d1ae1fc4a77895323c1596bc713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } } }
I20260927 22:56:25.751151 21406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d9f5d1ae1fc4a77895323c1596bc713. Latest consensus state: current_term: 1 leader_uuid: "8d9f5d1ae1fc4a77895323c1596bc713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9f5d1ae1fc4a77895323c1596bc713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } } }
I20260927 22:56:25.751775 21405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:25.751830 21406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:25.759392 21410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:25.763540 21410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:25.771679 21410 catalog_manager.cc:1347] Generated new cluster ID: c63efa9fe6e94b868eb15234e3552727
I20260927 22:56:25.771796 21410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:25.791222 21410 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:25.791697 21410 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:25.808722 21410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d9f5d1ae1fc4a77895323c1596bc713: Generated new TSK 0
I20260927 22:56:25.809561 21410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:25.826763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21327
W20260927 22:56:25.827862 21389 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54159
W20260927 22:56:25.828485 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21327: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:25.828632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 21327
2026-09-27T22:56:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1860 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:25 dist-test-slave-df53 krb5kdc[21425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:25 dist-test-slave-df53 krb5kdc[21425](info): set up 2 sockets
Sep 27 22:56:25 dist-test-slave-df53 krb5kdc[21425](info): commencing operation
krb5kdc: starting...
W20260927 22:56:26.913977 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:27 dist-test-slave-df53 krb5kdc[21425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:27.156652 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:27.473227 21438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:27.473495 21438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:27.473632 21438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:27.480932 21438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:27.481037 21438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:27.481087 21438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:27.481148 21438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:27.481190 21438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:27.490809 21438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:27.491979 21438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:27.493153 21438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:27.504433 21443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:27.504431 21444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:27.504362 21446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:27.505192 21438 server_base.cc:1034] running on GCE node
I20260927 22:56:27.506021 21438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:27.507596 21438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:27.508821 21438 hybrid_clock.cc:648] HybridClock initialized: now 1790549787508791 us; error 55 us; skew 500 ppm
I20260927 22:56:27.509191 21438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:27 dist-test-slave-df53 krb5kdc[21425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:27.512203 21438 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:27.513581 21438 webserver.cc:460] Webserver started at http://127.0.0.1:39077/ using document root <none> and password file <none>
I20260927 22:56:27.514253 21438 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:27.514350 21438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:27.514598 21438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:27.517376 21438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "1b13acbb849c4762b776909018ac2221"
format_stamp: "Formatted at 2026-09-27 22:56:27 on dist-test-slave-df53"
server_key: "77792fe5f6c31c1e113794d6ccc8ccc6"
server_key_iv: "3dd7737982e5367d2bac0d8c041cd3ba"
server_key_version: "encryptionkey@0"
I20260927 22:56:27.518181 21438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "1b13acbb849c4762b776909018ac2221"
format_stamp: "Formatted at 2026-09-27 22:56:27 on dist-test-slave-df53"
server_key: "77792fe5f6c31c1e113794d6ccc8ccc6"
server_key_iv: "3dd7737982e5367d2bac0d8c041cd3ba"
server_key_version: "encryptionkey@0"
I20260927 22:56:27.523084 21438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:56:27.526345 21453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:27.527668 21438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:56:27.527855 21438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "1b13acbb849c4762b776909018ac2221"
format_stamp: "Formatted at 2026-09-27 22:56:27 on dist-test-slave-df53"
server_key: "77792fe5f6c31c1e113794d6ccc8ccc6"
server_key_iv: "3dd7737982e5367d2bac0d8c041cd3ba"
server_key_version: "encryptionkey@0"
I20260927 22:56:27.528050 21438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:27.550477 21438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:27.554375 21438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:27.554680 21438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:27.573000 21438 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40861
I20260927 22:56:27.573124 21504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40861 every 8 connection(s)
I20260927 22:56:27.574895 21438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:27.577661 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 21438
I20260927 22:56:27.577858 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:27.578441 388 external_mini_cluster.cc:1428] Setting key 5d5305cfdce936343b1dbefce6e2e6ec
I20260927 22:56:27.579834 21505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:56:27 dist-test-slave-df53 krb5kdc[21425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:56:27.593325 21505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221: Bootstrap starting.
I20260927 22:56:27.597364 21505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:27.598606 21505 log.cc:824] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:27.601920 21505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221: No bootstrap required, opened a new log
I20260927 22:56:27.602553 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:27.580208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:38943 (local address 127.0.0.1:40861)
0927 22:56:27.580970 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:27.580985 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:27.581047 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:56:27.583134 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:27.583161 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:27.583678 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:27.584120 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:27.585410 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:27.587451 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:27.588728 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:27.589817 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:27.589996 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:27.594018 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:27.594028 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:27.594047 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:27.594094 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:27.598054 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:27.598785 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:27.598793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:27.598800 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:27.598992 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:27.599576 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:27.599579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:27.599581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:27.600179 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:27.600474 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:27.600876 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 22:56:27.601206 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":525,"thread_start_us":303,"threads_started":1}
I20260927 22:56:27.606432 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:27.610111 21505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13acbb849c4762b776909018ac2221" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } }
I20260927 22:56:27.610459 21505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:27.610595 21505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b13acbb849c4762b776909018ac2221, State: Initialized, Role: FOLLOWER
I20260927 22:56:27.611342 21505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13acbb849c4762b776909018ac2221" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } }
I20260927 22:56:27.611631 21505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:27.611784 21505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:27.611968 21505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:27.614898 21505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13acbb849c4762b776909018ac2221" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } }
I20260927 22:56:27.615430 21505 leader_election.cc:302] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b13acbb849c4762b776909018ac2221; no voters:
I20260927 22:56:27.616091 21505 leader_election.cc:288] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:27.616223 21515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:27.618309 21515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [term 1 LEADER]: Becoming Leader. State: Replica: 1b13acbb849c4762b776909018ac2221, State: Running, Role: LEADER
I20260927 22:56:27.618393 21505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:27.618824 21515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13acbb849c4762b776909018ac2221" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } }
I20260927 22:56:27.627727 21516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b13acbb849c4762b776909018ac2221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13acbb849c4762b776909018ac2221" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } } }
I20260927 22:56:27.627776 21517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b13acbb849c4762b776909018ac2221. Latest consensus state: current_term: 1 leader_uuid: "1b13acbb849c4762b776909018ac2221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b13acbb849c4762b776909018ac2221" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } } }
I20260927 22:56:27.628091 21516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:27.628142 21517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:27.630250 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:27.610723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:32783 (local address 127.0.0.1:40861)
0927 22:56:27.610919 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:27.610926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:27.611249 (+ 323us) server_negotiation.cc:407] Connection header received
0927 22:56:27.611412 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:27.611416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:27.611482 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:27.611587 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:27.613396 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:27.614253 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:27.616086 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:27.616954 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:27.617196 (+ 242us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:27.620556 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:27.620565 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:27.620574 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:27.620624 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:27.625803 (+ 5179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:27.626936 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:27.626943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:27.626947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:27.627221 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:27.627872 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:27.627879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:27.627884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:27.628284 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:27.629507 (+ 1223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:27.629715 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 22:56:27.629892 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 22:56:27.636473 21529 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:56:27.636608 21529 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:56:27.636767 21530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:27.639075 21530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:27.648065 21530 catalog_manager.cc:1347] Generated new cluster ID: 9433bda262b84f789298e233af5b3c83
I20260927 22:56:27.648176 21530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:27.665663 21530 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:27.666870 21530 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:27.676360 21530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b13acbb849c4762b776909018ac2221: Generated new TSK 0
I20260927 22:56:27.677102 21530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:27.688431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21438
I20260927 22:56:27.690143 21500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:27.780021 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 21438
W20260927 22:56:27.796669 21423 connection.cc:535] client connection to 127.0.0.1:40861 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40861): Connection reset by peer (error 104)
2026-09-27T22:56:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:27 dist-test-slave-df53 krb5kdc[21537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:27 dist-test-slave-df53 krb5kdc[21537](info): set up 2 sockets
Sep 27 22:56:27 dist-test-slave-df53 krb5kdc[21537](info): commencing operation
krb5kdc: starting...
W20260927 22:56:28.870018 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:29 dist-test-slave-df53 krb5kdc[21537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:29.103132 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:29.422173 21550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:29.422431 21550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:29.422716 21550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:29.429776 21550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:29.429869 21550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:29.429975 21550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:29.430084 21550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:29.430166 21550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:29.439452 21550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:29.440600 21550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:29.441839 21550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:29.453365 21556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:29.453333 21558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:29.453888 21555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:29.454303 21550 server_base.cc:1034] running on GCE node
I20260927 22:56:29.455000 21550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:29.456243 21550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:29.457460 21550 hybrid_clock.cc:648] HybridClock initialized: now 1790549789457424 us; error 49 us; skew 500 ppm
I20260927 22:56:29.457825 21550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:29 dist-test-slave-df53 krb5kdc[21537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:29.460805 21550 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:29.462208 21550 webserver.cc:460] Webserver started at http://127.0.0.1:46175/ using document root <none> and password file <none>
I20260927 22:56:29.462841 21550 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:29.462977 21550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:29.463272 21550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:29.466066 21550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "e9cc809ca3bc4a50a9a6992c235ecb69"
format_stamp: "Formatted at 2026-09-27 22:56:29 on dist-test-slave-df53"
server_key: "7e911c05b99ca80784d021afad70d4b3"
server_key_iv: "7468d6f36425f5cdc33ca393362a300d"
server_key_version: "encryptionkey@0"
I20260927 22:56:29.466811 21550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "e9cc809ca3bc4a50a9a6992c235ecb69"
format_stamp: "Formatted at 2026-09-27 22:56:29 on dist-test-slave-df53"
server_key: "7e911c05b99ca80784d021afad70d4b3"
server_key_iv: "7468d6f36425f5cdc33ca393362a300d"
server_key_version: "encryptionkey@0"
I20260927 22:56:29.472361 21550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:56:29.475682 21565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:29.476869 21550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:56:29.477037 21550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "e9cc809ca3bc4a50a9a6992c235ecb69"
format_stamp: "Formatted at 2026-09-27 22:56:29 on dist-test-slave-df53"
server_key: "7e911c05b99ca80784d021afad70d4b3"
server_key_iv: "7468d6f36425f5cdc33ca393362a300d"
server_key_version: "encryptionkey@0"
I20260927 22:56:29.477222 21550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:29.493894 21550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:29.497668 21550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:29.497941 21550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:29.515803 21550 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41263
I20260927 22:56:29.515839 21616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41263 every 8 connection(s)
I20260927 22:56:29.517107 21550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:29.520867 21617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:29.524780 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 21550
I20260927 22:56:29.524966 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:29.525499 388 external_mini_cluster.cc:1428] Setting key 54bb362f93b6822daefa0b85875afe99
I20260927 22:56:29.533342 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69: Bootstrap starting.
I20260927 22:56:29.536764 21617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:29.537942 21617 log.cc:824] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:29.538287 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:29.527232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:46369 (local address 127.0.0.1:41263)
0927 22:56:29.527937 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:29.527948 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:29.527987 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:56:29.529528 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:29.529549 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:29.530006 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:29.530323 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:29.530872 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:29.532160 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:29.533965 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:29.534594 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:29.534723 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:29.536017 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:29.536023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:29.536035 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:29.536526 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:29.536599 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:29.536624 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:29.536738 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:29.537091 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 22:56:29.537278 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":320,"threads_started":1}
I20260927 22:56:29.542019 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69: No bootstrap required, opened a new log
I20260927 22:56:29.542423 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:29.552136 21617 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cc809ca3bc4a50a9a6992c235ecb69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41263 } }
I20260927 22:56:29.552631 21617 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:29.552739 21617 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9cc809ca3bc4a50a9a6992c235ecb69, State: Initialized, Role: FOLLOWER
I20260927 22:56:29.553380 21617 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cc809ca3bc4a50a9a6992c235ecb69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41263 } }
I20260927 22:56:29.553668 21617 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:29.553820 21617 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:29.554028 21617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:29.554111 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:29.547550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:52015 (local address 127.0.0.1:41263)
0927 22:56:29.547722 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:29.547729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:29.548079 (+ 350us) server_negotiation.cc:407] Connection header received
0927 22:56:29.548199 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:29.548204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:29.548291 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:29.548404 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:29.548834 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:29.550019 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:29.551459 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:29.552076 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:29.552218 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:29.553237 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:29.553243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:29.553248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:29.553272 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:29.553324 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:29.553340 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:29.553464 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:29.553704 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 22:56:29.553755 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:56:29.556507 21617 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cc809ca3bc4a50a9a6992c235ecb69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41263 } }
I20260927 22:56:29.556989 21617 leader_election.cc:302] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9cc809ca3bc4a50a9a6992c235ecb69; no voters:
I20260927 22:56:29.557536 21617 leader_election.cc:288] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:29.557726 21627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:29.559617 21617 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:29.561874 21627 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [term 1 LEADER]: Becoming Leader. State: Replica: e9cc809ca3bc4a50a9a6992c235ecb69, State: Running, Role: LEADER
I20260927 22:56:29.562563 21627 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cc809ca3bc4a50a9a6992c235ecb69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41263 } }
I20260927 22:56:29.565692 21629 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9cc809ca3bc4a50a9a6992c235ecb69. Latest consensus state: current_term: 1 leader_uuid: "e9cc809ca3bc4a50a9a6992c235ecb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cc809ca3bc4a50a9a6992c235ecb69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41263 } } }
I20260927 22:56:29.566068 21629 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:29.566303 21628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9cc809ca3bc4a50a9a6992c235ecb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cc809ca3bc4a50a9a6992c235ecb69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41263 } } }
I20260927 22:56:29.567444 21628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69 [sys.catalog]: This master's current role is: LEADER
W20260927 22:56:29.570647 21641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:56:29.570794 21641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:56:29.570932 21643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:29.573005 21643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:29.582399 21643 catalog_manager.cc:1347] Generated new cluster ID: 9c0b9c2a0e0a4033b62b50c867af1593
I20260927 22:56:29.582497 21643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:29.597158 21643 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:29.598434 21643 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:29.609090 21643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9cc809ca3bc4a50a9a6992c235ecb69: Generated new TSK 0
I20260927 22:56:29.609907 21643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:29.623394 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21550
W20260927 22:56:29.624882 21612 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.117:46369
W20260927 22:56:29.625581 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21550: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:29.625753 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 21550
W20260927 22:56:29.643584 21535 connection.cc:535] client connection to 127.0.0.1:41263 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41263): Connection reset by peer (error 104)
2026-09-27T22:56:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1847 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:29 dist-test-slave-df53 krb5kdc[21648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:29 dist-test-slave-df53 krb5kdc[21648](info): set up 2 sockets
Sep 27 22:56:29 dist-test-slave-df53 krb5kdc[21648](info): commencing operation
krb5kdc: starting...
W20260927 22:56:30.733902 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:30 dist-test-slave-df53 krb5kdc[21648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:30.968443 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:31.288380 21661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:31.288703 21661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:31.288857 21661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:31.296171 21661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:31.296322 21661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:31.296422 21661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:31.296547 21661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:31.296625 21661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:31.306041 21661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:31.307232 21661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:31.308460 21661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:31.319777 21666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:31.319841 21667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:31.319916 21669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:31.321112 21661 server_base.cc:1034] running on GCE node
I20260927 22:56:31.321944 21661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:31.323277 21661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:31.324486 21661 hybrid_clock.cc:648] HybridClock initialized: now 1790549791324441 us; error 29 us; skew 500 ppm
I20260927 22:56:31.324774 21661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:31 dist-test-slave-df53 krb5kdc[21648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:31.331854 21661 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:31.333396 21661 webserver.cc:460] Webserver started at http://127.0.0.1:41265/ using document root <none> and password file <none>
I20260927 22:56:31.334138 21661 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:31.334270 21661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:31.334585 21661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:31.337460 21661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "2ac589c6d1aa453e88fb74bc6af63af3"
format_stamp: "Formatted at 2026-09-27 22:56:31 on dist-test-slave-df53"
server_key: "278d6674f2cb9d4ba39d9a9be07d17db"
server_key_iv: "691a6ca440d11718614ab07d277be801"
server_key_version: "encryptionkey@0"
I20260927 22:56:31.338253 21661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "2ac589c6d1aa453e88fb74bc6af63af3"
format_stamp: "Formatted at 2026-09-27 22:56:31 on dist-test-slave-df53"
server_key: "278d6674f2cb9d4ba39d9a9be07d17db"
server_key_iv: "691a6ca440d11718614ab07d277be801"
server_key_version: "encryptionkey@0"
I20260927 22:56:31.343046 21661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:56:31.346208 21676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:31.347429 21661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:56:31.347607 21661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "2ac589c6d1aa453e88fb74bc6af63af3"
format_stamp: "Formatted at 2026-09-27 22:56:31 on dist-test-slave-df53"
server_key: "278d6674f2cb9d4ba39d9a9be07d17db"
server_key_iv: "691a6ca440d11718614ab07d277be801"
server_key_version: "encryptionkey@0"
I20260927 22:56:31.347795 21661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:31.353220 21661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:31.356972 21661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:31.357262 21661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:31.375392 21661 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36661
I20260927 22:56:31.375438 21727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36661 every 8 connection(s)
I20260927 22:56:31.376682 21661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:31.379763 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 21661
I20260927 22:56:31.379943 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:31.380491 388 external_mini_cluster.cc:1428] Setting key 0da74c5ed8e1b76189b7b0b1ca573df1
I20260927 22:56:31.380533 21728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:31.387984 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:31.382211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:36857 (local address 127.0.0.1:36661)
0927 22:56:31.382954 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:31.382965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:31.383007 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:56:31.384914 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:31.384933 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:31.385244 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:31.385557 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:31.386572 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:31.386578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:31.386589 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:31.386603 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:31.386662 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:31.386680 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:31.386837 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:31.386946 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 22:56:31.387153 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":337,"threads_started":1}
I20260927 22:56:31.391331 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:31.393530 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3: Bootstrap starting.
I20260927 22:56:31.397193 21728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:31.398327 21728 log.cc:824] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:31.398790 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:31.396235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:39993 (local address 127.0.0.1:36661)
0927 22:56:31.396403 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:31.396410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:31.396944 (+ 534us) server_negotiation.cc:407] Connection header received
0927 22:56:31.397057 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:31.397065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:31.397128 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:31.397243 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:31.398142 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:31.398151 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:31.398156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:31.398174 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:31.398212 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:31.398230 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:31.398347 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:31.398420 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 22:56:31.398473 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:56:31.401055 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3: No bootstrap required, opened a new log
I20260927 22:56:31.407968 21728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac589c6d1aa453e88fb74bc6af63af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } }
I20260927 22:56:31.408314 21728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:31.408428 21728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac589c6d1aa453e88fb74bc6af63af3, State: Initialized, Role: FOLLOWER
I20260927 22:56:31.408950 21728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac589c6d1aa453e88fb74bc6af63af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } }
I20260927 22:56:31.409176 21728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:31.409322 21728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:31.409480 21728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:31.411510 21728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac589c6d1aa453e88fb74bc6af63af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } }
I20260927 22:56:31.411962 21728 leader_election.cc:302] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ac589c6d1aa453e88fb74bc6af63af3; no voters:
I20260927 22:56:31.412513 21728 leader_election.cc:288] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:31.412690 21738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:31.413385 21738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [term 1 LEADER]: Becoming Leader. State: Replica: 2ac589c6d1aa453e88fb74bc6af63af3, State: Running, Role: LEADER
I20260927 22:56:31.414047 21738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac589c6d1aa453e88fb74bc6af63af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } }
I20260927 22:56:31.414973 21728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:31.416760 21739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ac589c6d1aa453e88fb74bc6af63af3. Latest consensus state: current_term: 1 leader_uuid: "2ac589c6d1aa453e88fb74bc6af63af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac589c6d1aa453e88fb74bc6af63af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } } }
I20260927 22:56:31.417147 21740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ac589c6d1aa453e88fb74bc6af63af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac589c6d1aa453e88fb74bc6af63af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } } }
I20260927 22:56:31.417500 21739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:31.418054 21740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:31.419348 21744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:31.425228 21744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:31.433753 21744 catalog_manager.cc:1347] Generated new cluster ID: ce732ee770ec4d92a33c7d6443b88246
I20260927 22:56:31.433858 21744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:31.452338 21744 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:31.452739 21744 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:31.461535 21744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ac589c6d1aa453e88fb74bc6af63af3: Generated new TSK 0
I20260927 22:56:31.462598 21744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:31.468963 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21661
W20260927 22:56:31.470213 21723 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.117:36857
W20260927 22:56:31.470894 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21661: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:31.471046 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 21661
2026-09-27T22:56:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:31 dist-test-slave-df53 krb5kdc[21759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:31 dist-test-slave-df53 krb5kdc[21759](info): set up 2 sockets
Sep 27 22:56:31 dist-test-slave-df53 krb5kdc[21759](info): commencing operation
krb5kdc: starting...
W20260927 22:56:32.565956 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:32 dist-test-slave-df53 krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:32.798341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:33.113859 21772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:33.114118 21772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:33.114207 21772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:33.121052 21772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:33.121146 21772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:33.121191 21772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:33.121250 21772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:33.121294 21772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:33.130537 21772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:33.131683 21772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:33.132864 21772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:33.144075 21780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:33.144093 21778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:33.144122 21777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:33.145812 21772 server_base.cc:1034] running on GCE node
I20260927 22:56:33.146477 21772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:33.147737 21772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:33.148928 21772 hybrid_clock.cc:648] HybridClock initialized: now 1790549793148879 us; error 46 us; skew 500 ppm
I20260927 22:56:33.149222 21772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:33 dist-test-slave-df53 krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:33.152206 21772 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:33.153716 21772 webserver.cc:460] Webserver started at http://127.0.0.1:37819/ using document root <none> and password file <none>
I20260927 22:56:33.154363 21772 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:33.154495 21772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:33.154803 21772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:33.157688 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "444e7e7f27bb4719859d4cdcb2fd60e9"
format_stamp: "Formatted at 2026-09-27 22:56:33 on dist-test-slave-df53"
server_key: "9db7e7ed643b30e2aced4552650df083"
server_key_iv: "5415f51a84878d43d926e09dadc99bc9"
server_key_version: "encryptionkey@0"
I20260927 22:56:33.158425 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "444e7e7f27bb4719859d4cdcb2fd60e9"
format_stamp: "Formatted at 2026-09-27 22:56:33 on dist-test-slave-df53"
server_key: "9db7e7ed643b30e2aced4552650df083"
server_key_iv: "5415f51a84878d43d926e09dadc99bc9"
server_key_version: "encryptionkey@0"
I20260927 22:56:33.163254 21772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:56:33.166517 21787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:33.167730 21772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:33.167905 21772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "444e7e7f27bb4719859d4cdcb2fd60e9"
format_stamp: "Formatted at 2026-09-27 22:56:33 on dist-test-slave-df53"
server_key: "9db7e7ed643b30e2aced4552650df083"
server_key_iv: "5415f51a84878d43d926e09dadc99bc9"
server_key_version: "encryptionkey@0"
I20260927 22:56:33.168099 21772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:33.203258 21772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:33.207185 21772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:33.207459 21772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:33.225764 21772 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42113
I20260927 22:56:33.225785 21838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42113 every 8 connection(s)
I20260927 22:56:33.227056 21772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:33.230100 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 21772
I20260927 22:56:33.230281 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:33.230818 388 external_mini_cluster.cc:1428] Setting key b79dcdc74e111ac886c76f784f27daa9
I20260927 22:56:33.230978 21839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:33.245066 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9: Bootstrap starting.
Sep 27 22:56:33 dist-test-slave-df53 krb5kdc[21759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:56:33.248106 21839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:33.249127 21839 log.cc:824] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:33.251873 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9: No bootstrap required, opened a new log
I20260927 22:56:33.253741 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:33.232488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:35763 (local address 127.0.0.1:42113)
0927 22:56:33.233235 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:33.233246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:33.233283 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:56:33.235480 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:33.235501 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:33.235880 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:33.236195 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:33.237879 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:33.239124 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:33.240900 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:33.241559 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:33.241748 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:33.246481 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:33.246488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:33.246501 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:33.246537 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:33.249678 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:33.250485 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:33.250501 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:33.250507 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:33.250660 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:33.251129 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:33.251132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:33.251134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:33.251632 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:33.251944 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:33.252457 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 22:56:33.252753 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":342,"threads_started":1}
I20260927 22:56:33.257361 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:33.259919 21839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444e7e7f27bb4719859d4cdcb2fd60e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260927 22:56:33.260329 21839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:33.260468 21839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444e7e7f27bb4719859d4cdcb2fd60e9, State: Initialized, Role: FOLLOWER
I20260927 22:56:33.261026 21839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444e7e7f27bb4719859d4cdcb2fd60e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260927 22:56:33.261264 21839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:33.261397 21839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:33.261632 21839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:33.264169 21839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444e7e7f27bb4719859d4cdcb2fd60e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260927 22:56:33.264720 21839 leader_election.cc:302] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 444e7e7f27bb4719859d4cdcb2fd60e9; no voters:
I20260927 22:56:33.265376 21839 leader_election.cc:288] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:33.265533 21849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:33.270433 21849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [term 1 LEADER]: Becoming Leader. State: Replica: 444e7e7f27bb4719859d4cdcb2fd60e9, State: Running, Role: LEADER
I20260927 22:56:33.271129 21849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444e7e7f27bb4719859d4cdcb2fd60e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260927 22:56:33.271950 21839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:33.274315 21850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "444e7e7f27bb4719859d4cdcb2fd60e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444e7e7f27bb4719859d4cdcb2fd60e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } } }
I20260927 22:56:33.274739 21850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:33.275255 21851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 444e7e7f27bb4719859d4cdcb2fd60e9. Latest consensus state: current_term: 1 leader_uuid: "444e7e7f27bb4719859d4cdcb2fd60e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444e7e7f27bb4719859d4cdcb2fd60e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } } }
I20260927 22:56:33.275569 21851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:33.278573 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:33.261574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:51419 (local address 127.0.0.1:42113)
0927 22:56:33.261765 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:33.261771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:33.262286 (+ 515us) server_negotiation.cc:407] Connection header received
0927 22:56:33.262472 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:33.262477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:33.262565 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:33.262679 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:33.264288 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:33.265256 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:33.267064 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:33.267529 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:33.267675 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:33.270466 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:33.270477 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:33.270484 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:33.270530 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:33.274341 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:33.275261 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:33.275270 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:33.275276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:33.275482 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:33.276233 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:33.276240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:33.276245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:33.276678 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:33.277007 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:33.277849 (+ 842us) server_negotiation.cc:299] Negotiation successful
0927 22:56:33.278089 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:56:33.286494 21854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:33.292474 21854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:33.301525 21854 catalog_manager.cc:1347] Generated new cluster ID: de92b87c594441f1a815236efe7113f1
I20260927 22:56:33.301682 21854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:33.332456 21854 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:33.333806 21854 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:33.347047 21854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 444e7e7f27bb4719859d4cdcb2fd60e9: Generated new TSK 0
I20260927 22:56:33.347868 21854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:33.360376 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21772
I20260927 22:56:33.361745 21834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:33.454653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 21772
W20260927 22:56:33.471401 21757 connection.cc:535] client connection to 127.0.0.1:42113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42113): Connection reset by peer (error 104)
2026-09-27T22:56:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:33 dist-test-slave-df53 krb5kdc[21872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:33 dist-test-slave-df53 krb5kdc[21872](info): set up 2 sockets
Sep 27 22:56:33 dist-test-slave-df53 krb5kdc[21872](info): commencing operation
krb5kdc: starting...
W20260927 22:56:34.561944 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:34 dist-test-slave-df53 krb5kdc[21872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:34.792104 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:35.105645 21885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:35.105965 21885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:35.106124 21885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:35.113149 21885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:35.113296 21885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:35.113391 21885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:35.113497 21885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:35.113581 21885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:35.122967 21885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:35.124145 21885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:35.125427 21885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:35.136969 21890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:35.136996 21893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:35.137156 21891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:35.138106 21885 server_base.cc:1034] running on GCE node
I20260927 22:56:35.138829 21885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:35.140089 21885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:35.141314 21885 hybrid_clock.cc:648] HybridClock initialized: now 1790549795141279 us; error 45 us; skew 500 ppm
I20260927 22:56:35.141709 21885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:35 dist-test-slave-df53 krb5kdc[21872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:35.144629 21885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:35.146080 21885 webserver.cc:460] Webserver started at http://127.0.0.1:45739/ using document root <none> and password file <none>
I20260927 22:56:35.146721 21885 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:35.146865 21885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:35.147147 21885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:35.149979 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "9f81dbc54ccc482c88b47c3aafde23cd"
format_stamp: "Formatted at 2026-09-27 22:56:35 on dist-test-slave-df53"
server_key: "93ab68c6a29db0500dd59dc8ef01544b"
server_key_iv: "cc589c9f446f7fe772944c2619feb5e4"
server_key_version: "encryptionkey@0"
I20260927 22:56:35.150695 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "9f81dbc54ccc482c88b47c3aafde23cd"
format_stamp: "Formatted at 2026-09-27 22:56:35 on dist-test-slave-df53"
server_key: "93ab68c6a29db0500dd59dc8ef01544b"
server_key_iv: "cc589c9f446f7fe772944c2619feb5e4"
server_key_version: "encryptionkey@0"
I20260927 22:56:35.155447 21885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 22:56:35.158564 21900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:35.159729 21885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:56:35.159902 21885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "9f81dbc54ccc482c88b47c3aafde23cd"
format_stamp: "Formatted at 2026-09-27 22:56:35 on dist-test-slave-df53"
server_key: "93ab68c6a29db0500dd59dc8ef01544b"
server_key_iv: "cc589c9f446f7fe772944c2619feb5e4"
server_key_version: "encryptionkey@0"
I20260927 22:56:35.160082 21885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:35.177688 21885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:35.181416 21885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:35.181733 21885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:35.200133 21885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44697
I20260927 22:56:35.200212 21951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44697 every 8 connection(s)
I20260927 22:56:35.201450 21885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:35.202991 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 21885
I20260927 22:56:35.203208 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:35.203610 388 external_mini_cluster.cc:1428] Setting key b98142ec88b79a7a27ffb7e2c52b7e61
I20260927 22:56:35.205555 21952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:35.216437 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:35.205214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:46827 (local address 127.0.0.1:44697)
0927 22:56:35.206031 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:35.206041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:35.206080 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:56:35.207617 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:35.207636 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:35.207991 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:35.208295 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:35.208894 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:35.210214 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:35.212179 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:35.212823 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:35.212943 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:35.214262 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:35.214270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:35.214282 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:35.214806 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:35.214874 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:35.214899 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:35.215027 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:35.215344 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 22:56:35.215512 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":432,"threads_started":1}
I20260927 22:56:35.217973 21952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd: Bootstrap starting.
I20260927 22:56:35.220181 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:35.221095 21952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:35.222101 21952 log.cc:824] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:35.224714 21952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd: No bootstrap required, opened a new log
I20260927 22:56:35.231145 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:35.224600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:51765 (local address 127.0.0.1:44697)
0927 22:56:35.224808 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:35.224816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:35.225196 (+ 380us) server_negotiation.cc:407] Connection header received
0927 22:56:35.225394 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:35.225397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:35.225451 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:35.225526 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:35.226143 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:35.226886 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:35.228796 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:35.229198 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:35.229281 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:35.230406 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:35.230410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:35.230413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:35.230427 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:35.230479 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:35.230493 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:35.230576 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:35.230771 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 22:56:35.230836 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:56:35.233856 21952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f81dbc54ccc482c88b47c3aafde23cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44697 } }
I20260927 22:56:35.234289 21952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:35.234391 21952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f81dbc54ccc482c88b47c3aafde23cd, State: Initialized, Role: FOLLOWER
I20260927 22:56:35.234901 21952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f81dbc54ccc482c88b47c3aafde23cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44697 } }
I20260927 22:56:35.235105 21952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:35.235199 21952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:35.235318 21952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:35.237298 21952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f81dbc54ccc482c88b47c3aafde23cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44697 } }
I20260927 22:56:35.237812 21952 leader_election.cc:302] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f81dbc54ccc482c88b47c3aafde23cd; no voters:
I20260927 22:56:35.238368 21952 leader_election.cc:288] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:35.238631 21962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:35.239382 21962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [term 1 LEADER]: Becoming Leader. State: Replica: 9f81dbc54ccc482c88b47c3aafde23cd, State: Running, Role: LEADER
I20260927 22:56:35.239898 21962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f81dbc54ccc482c88b47c3aafde23cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44697 } }
I20260927 22:56:35.240679 21952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:35.242856 21964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f81dbc54ccc482c88b47c3aafde23cd. Latest consensus state: current_term: 1 leader_uuid: "9f81dbc54ccc482c88b47c3aafde23cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f81dbc54ccc482c88b47c3aafde23cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44697 } } }
I20260927 22:56:35.243310 21964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:35.243055 21963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f81dbc54ccc482c88b47c3aafde23cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f81dbc54ccc482c88b47c3aafde23cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44697 } } }
I20260927 22:56:35.243481 21963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:35.246048 21967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:35.250993 21967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:35.259479 21967 catalog_manager.cc:1347] Generated new cluster ID: 9ae0dd6d7a8d4bc694bb0d3862e00c03
I20260927 22:56:35.259573 21967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:35.279771 21967 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:35.281061 21967 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:35.287133 21967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f81dbc54ccc482c88b47c3aafde23cd: Generated new TSK 0
I20260927 22:56:35.287875 21967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:35.299584 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21885
W20260927 22:56:35.300652 21947 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.117:46827
W20260927 22:56:35.301380 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21885: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:35.301556 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 21885
W20260927 22:56:35.317858 21870 connection.cc:535] client connection to 127.0.0.1:44697 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44697): Connection reset by peer (error 104)
2026-09-27T22:56:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1850 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:35 dist-test-slave-df53 krb5kdc[21983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:35 dist-test-slave-df53 krb5kdc[21983](info): set up 2 sockets
Sep 27 22:56:35 dist-test-slave-df53 krb5kdc[21983](info): commencing operation
krb5kdc: starting...
W20260927 22:56:36.401961 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:36 dist-test-slave-df53 krb5kdc[21983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:56:36.634171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:36.952992 21996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:36.953305 21996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:36.953485 21996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:36.961027 21996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:36.961180 21996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:36.961263 21996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:36.961346 21996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:36.961407 21996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:36.971009 21996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33073
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:36.972234 21996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:36.973513 21996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:36.984898 22001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:36.984965 22004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:36.984988 22002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:36.985622 21996 server_base.cc:1034] running on GCE node
I20260927 22:56:36.986406 21996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:36.987705 21996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:36.988929 21996 hybrid_clock.cc:648] HybridClock initialized: now 1790549796988888 us; error 54 us; skew 500 ppm
I20260927 22:56:36.989269 21996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:36 dist-test-slave-df53 krb5kdc[21983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:36.997284 21996 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:56:36.998780 21996 webserver.cc:460] Webserver started at http://127.0.0.1:33495/ using document root <none> and password file <none>
I20260927 22:56:36.999394 21996 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:36.999490 21996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:36.999729 21996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:37.002584 21996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "a37a6b7dd5e843f5b1a602b1a906e444"
format_stamp: "Formatted at 2026-09-27 22:56:36 on dist-test-slave-df53"
server_key: "207545ec70fa1487e9b610427e1b21cd"
server_key_iv: "cdc9e2f54abfe4a868cf0d453eaf6310"
server_key_version: "encryptionkey@0"
I20260927 22:56:37.003355 21996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "a37a6b7dd5e843f5b1a602b1a906e444"
format_stamp: "Formatted at 2026-09-27 22:56:36 on dist-test-slave-df53"
server_key: "207545ec70fa1487e9b610427e1b21cd"
server_key_iv: "cdc9e2f54abfe4a868cf0d453eaf6310"
server_key_version: "encryptionkey@0"
I20260927 22:56:37.008155 21996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:56:37.011380 22011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:37.012619 21996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:56:37.012791 21996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "a37a6b7dd5e843f5b1a602b1a906e444"
format_stamp: "Formatted at 2026-09-27 22:56:36 on dist-test-slave-df53"
server_key: "207545ec70fa1487e9b610427e1b21cd"
server_key_iv: "cdc9e2f54abfe4a868cf0d453eaf6310"
server_key_version: "encryptionkey@0"
I20260927 22:56:37.012996 21996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:37.018448 21996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:37.022423 21996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:37.022722 21996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:37.041247 21996 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33073
I20260927 22:56:37.041323 22062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33073 every 8 connection(s)
I20260927 22:56:37.042584 21996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:37.043681 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 21996
I20260927 22:56:37.043869 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:37.044261 388 external_mini_cluster.cc:1428] Setting key 0a5f6fc65ad03eadc39c3a6854310be7
I20260927 22:56:37.046834 22063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:37.051450 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:37.045892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:40879 (local address 127.0.0.1:33073)
0927 22:56:37.046591 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:37.046603 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:37.046639 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:56:37.048280 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:37.048300 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:37.048604 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:37.048967 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:37.049972 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:37.049977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:37.049989 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:37.050001 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:37.050067 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:37.050094 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:37.050346 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:37.050474 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 22:56:37.050642 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":308,"threads_started":1}
I20260927 22:56:37.054904 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:37.058030 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444: Bootstrap starting.
I20260927 22:56:37.061316 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:37.058937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:41465 (local address 127.0.0.1:33073)
0927 22:56:37.059076 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:37.059106 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:37.059525 (+ 419us) server_negotiation.cc:407] Connection header received
0927 22:56:37.059701 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:37.059706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:37.059755 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:37.059865 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:37.060712 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:37.060717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:37.060721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:37.060738 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:37.060783 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:37.060796 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:37.060908 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:37.061009 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 22:56:37.061063 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:56:37.061384 22063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:37.062546 22063 log.cc:824] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:37.065129 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444: No bootstrap required, opened a new log
I20260927 22:56:37.072418 22063 raft_consensus.cc:348] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37a6b7dd5e843f5b1a602b1a906e444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33073 } }
I20260927 22:56:37.072765 22063 raft_consensus.cc:374] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:37.072883 22063 raft_consensus.cc:729] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37a6b7dd5e843f5b1a602b1a906e444, State: Initialized, Role: FOLLOWER
I20260927 22:56:37.073468 22063 consensus_queue.cc:260] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37a6b7dd5e843f5b1a602b1a906e444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33073 } }
I20260927 22:56:37.073750 22063 raft_consensus.cc:388] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:37.073894 22063 raft_consensus.cc:482] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:37.074060 22063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:37.076481 22063 raft_consensus.cc:504] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37a6b7dd5e843f5b1a602b1a906e444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33073 } }
I20260927 22:56:37.076946 22063 leader_election.cc:302] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a37a6b7dd5e843f5b1a602b1a906e444; no voters:
I20260927 22:56:37.077495 22063 leader_election.cc:288] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:37.077714 22073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:37.078347 22073 raft_consensus.cc:686] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [term 1 LEADER]: Becoming Leader. State: Replica: a37a6b7dd5e843f5b1a602b1a906e444, State: Running, Role: LEADER
I20260927 22:56:37.078896 22073 consensus_queue.cc:237] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37a6b7dd5e843f5b1a602b1a906e444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33073 } }
I20260927 22:56:37.079494 22063 sys_catalog.cc:564] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:37.081524 22074 sys_catalog.cc:455] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a37a6b7dd5e843f5b1a602b1a906e444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37a6b7dd5e843f5b1a602b1a906e444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33073 } } }
I20260927 22:56:37.081795 22075 sys_catalog.cc:455] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a37a6b7dd5e843f5b1a602b1a906e444. Latest consensus state: current_term: 1 leader_uuid: "a37a6b7dd5e843f5b1a602b1a906e444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a37a6b7dd5e843f5b1a602b1a906e444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33073 } } }
I20260927 22:56:37.082053 22074 sys_catalog.cc:458] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:37.082057 22075 sys_catalog.cc:458] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:37.082901 22080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:37.092566 22080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:37.101395 22080 catalog_manager.cc:1347] Generated new cluster ID: 4c53a1269fa941c2b702fc9688ba4739
I20260927 22:56:37.101564 22080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:37.116029 22080 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:37.116464 22080 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:37.126936 22080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a37a6b7dd5e843f5b1a602b1a906e444: Generated new TSK 0
I20260927 22:56:37.127609 22080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:37.134276 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21996
W20260927 22:56:37.135717 22058 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.117:40879
W20260927 22:56:37.136368 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 21996: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:37.136498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 21996
2026-09-27T22:56:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1830 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:37 dist-test-slave-df53 krb5kdc[22094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:37 dist-test-slave-df53 krb5kdc[22094](info): set up 2 sockets
Sep 27 22:56:37 dist-test-slave-df53 krb5kdc[22094](info): commencing operation
krb5kdc: starting...
W20260927 22:56:38.229872 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:38 dist-test-slave-df53 krb5kdc[22094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:56:38.453867 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42101
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:38.769335 22107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:38.769711 22107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:38.769879 22107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:38.777123 22107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:38.777217 22107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:38.777328 22107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:38.777428 22107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:38.777504 22107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:38.787070 22107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:38.788300 22107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:38.789657 22107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:38.801002 22113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:38.801028 22115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:38.801309 22112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:38.801764 22107 server_base.cc:1034] running on GCE node
I20260927 22:56:38.802542 22107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:38.803941 22107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:38.805167 22107 hybrid_clock.cc:648] HybridClock initialized: now 1790549798805132 us; error 50 us; skew 500 ppm
I20260927 22:56:38.805511 22107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:38 dist-test-slave-df53 krb5kdc[22094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:38.808477 22107 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:56:38.810034 22107 webserver.cc:460] Webserver started at http://127.0.97.62:37243/ using document root <none> and password file <none>
I20260927 22:56:38.810675 22107 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:38.810817 22107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:38.811098 22107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:38.814073 22107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "4382c1e666e9495186cef8e7ce3410ca"
format_stamp: "Formatted at 2026-09-27 22:56:38 on dist-test-slave-df53"
server_key: "0c386bd5b10fb8643f6f3cb76ca663f9"
server_key_iv: "0e614853b7acd25ec50e14dc1cd6d87a"
server_key_version: "encryptionkey@0"
I20260927 22:56:38.814831 22107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "4382c1e666e9495186cef8e7ce3410ca"
format_stamp: "Formatted at 2026-09-27 22:56:38 on dist-test-slave-df53"
server_key: "0c386bd5b10fb8643f6f3cb76ca663f9"
server_key_iv: "0e614853b7acd25ec50e14dc1cd6d87a"
server_key_version: "encryptionkey@0"
I20260927 22:56:38.819828 22107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:56:38.823072 22122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:38.824244 22107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:56:38.824426 22107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "4382c1e666e9495186cef8e7ce3410ca"
format_stamp: "Formatted at 2026-09-27 22:56:38 on dist-test-slave-df53"
server_key: "0c386bd5b10fb8643f6f3cb76ca663f9"
server_key_iv: "0e614853b7acd25ec50e14dc1cd6d87a"
server_key_version: "encryptionkey@0"
I20260927 22:56:38.824622 22107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:38.840855 22107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:38.844506 22107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:38.844821 22107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:38.862435 22107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42101
I20260927 22:56:38.862468 22173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42101 every 8 connection(s)
I20260927 22:56:38.863722 22107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:38.864135 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 22107
I20260927 22:56:38.864319 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:38.864909 388 external_mini_cluster.cc:1428] Setting key 261241ff9b25924e1545169d468c49d3
I20260927 22:56:38.868026 22174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:56:38 dist-test-slave-df53 krb5kdc[22094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:38.879942 22174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca: Bootstrap starting.
I20260927 22:56:38.883461 22174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:38.884445 22174 log.cc:824] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:38.887162 22174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca: No bootstrap required, opened a new log
I20260927 22:56:38.888047 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:38.866657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58395 (local address 127.0.97.62:42101)
0927 22:56:38.867379 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:38.867392 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:38.867437 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:56:38.869677 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:38.869702 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:38.870145 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:38.870596 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:38.871821 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:38.873987 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:38.875245 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:38.876227 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:38.876412 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:38.880313 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:38.880321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:38.880334 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:38.880373 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:38.883903 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:38.884646 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:38.884652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:38.884658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:38.884812 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:38.885299 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:38.885303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:38.885305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:38.885853 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:38.886167 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:38.886670 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 22:56:38.886965 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":305,"threads_started":1}
I20260927 22:56:38.891897 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:38.895408 22174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4382c1e666e9495186cef8e7ce3410ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } }
I20260927 22:56:38.895745 22174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:38.895882 22174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4382c1e666e9495186cef8e7ce3410ca, State: Initialized, Role: FOLLOWER
I20260927 22:56:38.896559 22174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4382c1e666e9495186cef8e7ce3410ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } }
I20260927 22:56:38.896850 22174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:38.897004 22174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:38.897229 22174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:38.899896 22174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4382c1e666e9495186cef8e7ce3410ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } }
I20260927 22:56:38.900352 22174 leader_election.cc:302] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4382c1e666e9495186cef8e7ce3410ca; no voters:
I20260927 22:56:38.900914 22174 leader_election.cc:288] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:38.901053 22184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:38.901731 22184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [term 1 LEADER]: Becoming Leader. State: Replica: 4382c1e666e9495186cef8e7ce3410ca, State: Running, Role: LEADER
I20260927 22:56:38.902361 22184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4382c1e666e9495186cef8e7ce3410ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } }
I20260927 22:56:38.903849 22174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:38.905292 22185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4382c1e666e9495186cef8e7ce3410ca. Latest consensus state: current_term: 1 leader_uuid: "4382c1e666e9495186cef8e7ce3410ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4382c1e666e9495186cef8e7ce3410ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } } }
I20260927 22:56:38.905684 22185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:38.910653 22186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4382c1e666e9495186cef8e7ce3410ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4382c1e666e9495186cef8e7ce3410ca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42101 } } }
I20260927 22:56:38.911494 22186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:38.914867 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:38.896031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58033 (local address 127.0.97.62:42101)
0927 22:56:38.896204 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:38.896212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:38.896625 (+ 413us) server_negotiation.cc:407] Connection header received
0927 22:56:38.896721 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:38.896726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:38.896796 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:38.896907 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:38.898427 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:38.899575 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:38.901032 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:38.901945 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:38.902153 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:38.905260 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:38.905268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:38.905280 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:38.905321 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:38.909853 (+ 4532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:38.911015 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:38.911024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:38.911030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:38.911308 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:38.911983 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:38.911991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:38.911996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:38.912478 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:38.912855 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:38.914193 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0927 22:56:38.914452 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:56:38.915071 22189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:38.920953 22189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:38.929332 22189 catalog_manager.cc:1347] Generated new cluster ID: d8f3988b41e9445aab3b5e07b9247f8d
I20260927 22:56:38.929433 22189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:38.947562 22189 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:38.949297 22189 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:38.960839 22189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4382c1e666e9495186cef8e7ce3410ca: Generated new TSK 0
I20260927 22:56:38.961791 22189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:38.969334 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22107
I20260927 22:56:38.970549 22169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:39.061937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 22107
W20260927 22:56:39.079074 22092 connection.cc:535] client connection to 127.0.97.62:42101 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:42101): Connection reset by peer (error 104)
2026-09-27T22:56:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:39 dist-test-slave-df53 krb5kdc[22206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:39 dist-test-slave-df53 krb5kdc[22206](info): set up 2 sockets
Sep 27 22:56:39 dist-test-slave-df53 krb5kdc[22206](info): commencing operation
krb5kdc: starting...
W20260927 22:56:40.154029 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:40 dist-test-slave-df53 krb5kdc[22206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:56:40.384472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39585
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:40.703150 22219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:40.703461 22219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:40.703635 22219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:40.710628 22219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:40.710769 22219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:40.710873 22219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:40.710981 22219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:40.711067 22219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:40.720513 22219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39585
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:40.721784 22219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:40.723043 22219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:40.734244 22224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:40.734305 22227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:40.734346 22225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:40.735785 22219 server_base.cc:1034] running on GCE node
I20260927 22:56:40.736461 22219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:40.737761 22219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:40.738981 22219 hybrid_clock.cc:648] HybridClock initialized: now 1790549800738943 us; error 47 us; skew 500 ppm
I20260927 22:56:40.739306 22219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:40 dist-test-slave-df53 krb5kdc[22206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:40.742250 22219 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:56:40.743486 22219 webserver.cc:460] Webserver started at http://127.0.97.62:38555/ using document root <none> and password file <none>
I20260927 22:56:40.744118 22219 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:40.744215 22219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:40.744449 22219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:40.747212 22219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "ef52370238d4468da75507228747f34c"
format_stamp: "Formatted at 2026-09-27 22:56:40 on dist-test-slave-df53"
server_key: "f6602cfb88e2ec26379d53dc4e98a5ea"
server_key_iv: "9c9f0d859c08c078c56d1f159e461e37"
server_key_version: "encryptionkey@0"
I20260927 22:56:40.747877 22219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef52370238d4468da75507228747f34c"
format_stamp: "Formatted at 2026-09-27 22:56:40 on dist-test-slave-df53"
server_key: "f6602cfb88e2ec26379d53dc4e98a5ea"
server_key_iv: "9c9f0d859c08c078c56d1f159e461e37"
server_key_version: "encryptionkey@0"
I20260927 22:56:40.752422 22219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:56:40.755761 22234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:40.757018 22219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:56:40.757180 22219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "ef52370238d4468da75507228747f34c"
format_stamp: "Formatted at 2026-09-27 22:56:40 on dist-test-slave-df53"
server_key: "f6602cfb88e2ec26379d53dc4e98a5ea"
server_key_iv: "9c9f0d859c08c078c56d1f159e461e37"
server_key_version: "encryptionkey@0"
I20260927 22:56:40.757371 22219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:40.784885 22219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:40.788851 22219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:40.789136 22219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:40.807852 22219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39585
I20260927 22:56:40.807940 22285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39585 every 8 connection(s)
I20260927 22:56:40.809244 22219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:40.812942 22286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:40.814944 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 22219
I20260927 22:56:40.815145 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:40.815587 388 external_mini_cluster.cc:1428] Setting key dc4a06d1a2c8c60c1db779f664b28fc0
I20260927 22:56:40.827167 22286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c: Bootstrap starting.
I20260927 22:56:40.830025 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:40.817389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46017 (local address 127.0.97.62:39585)
0927 22:56:40.818283 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:40.818303 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:40.818361 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:56:40.820019 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:40.820046 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:40.820570 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:40.820970 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:40.821446 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:40.823380 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:40.824712 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:40.825820 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:40.825998 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:40.827254 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:40.827264 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:40.827281 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:40.827963 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:40.828057 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:40.828088 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:40.828229 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:40.828509 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 22:56:40.828698 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":399,"threads_started":1}
I20260927 22:56:40.830950 22286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:40.832015 22286 log.cc:824] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:40.834193 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:40.834815 22286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c: No bootstrap required, opened a new log
I20260927 22:56:40.843119 22286 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef52370238d4468da75507228747f34c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } }
I20260927 22:56:40.843585 22286 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:40.843690 22286 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef52370238d4468da75507228747f34c, State: Initialized, Role: FOLLOWER
I20260927 22:56:40.844344 22286 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef52370238d4468da75507228747f34c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } }
I20260927 22:56:40.844645 22286 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:40.844846 22286 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:40.845122 22286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:40.845211 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:40.838566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60975 (local address 127.0.97.62:39585)
0927 22:56:40.838747 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:40.838754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:40.839118 (+ 364us) server_negotiation.cc:407] Connection header received
0927 22:56:40.839259 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:40.839266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:40.839351 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:40.839457 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:40.839953 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:40.841084 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:40.842507 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:40.843152 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:40.843276 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:40.844411 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:40.844417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:40.844423 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:40.844443 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:40.844502 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:40.844527 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:40.844662 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:40.844887 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 22:56:40.844941 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:56:40.847656 22286 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef52370238d4468da75507228747f34c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } }
I20260927 22:56:40.848217 22286 leader_election.cc:302] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef52370238d4468da75507228747f34c; no voters:
I20260927 22:56:40.848891 22286 leader_election.cc:288] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:40.849040 22296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:40.849867 22296 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [term 1 LEADER]: Becoming Leader. State: Replica: ef52370238d4468da75507228747f34c, State: Running, Role: LEADER
I20260927 22:56:40.850433 22296 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef52370238d4468da75507228747f34c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } }
I20260927 22:56:40.851055 22286 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:40.853447 22298 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef52370238d4468da75507228747f34c. Latest consensus state: current_term: 1 leader_uuid: "ef52370238d4468da75507228747f34c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef52370238d4468da75507228747f34c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } } }
I20260927 22:56:40.853937 22298 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:40.853832 22297 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef52370238d4468da75507228747f34c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef52370238d4468da75507228747f34c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } } }
I20260927 22:56:40.854130 22297 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:40.855494 22302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:40.860957 22302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:40.870224 22302 catalog_manager.cc:1347] Generated new cluster ID: 2bdb5b10f8c84afd95c1cdc249678414
I20260927 22:56:40.870332 22302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:40.906791 22302 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:40.908072 22302 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:40.914309 22302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef52370238d4468da75507228747f34c: Generated new TSK 0
I20260927 22:56:40.915117 22302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:40.926270 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22219
W20260927 22:56:40.927464 22281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46017
W20260927 22:56:40.928272 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22219: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:40.928432 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 22219
W20260927 22:56:40.945983 22204 connection.cc:535] client connection to 127.0.97.62:39585 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:39585): Connection reset by peer (error 104)
2026-09-27T22:56:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1866 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:41 dist-test-slave-df53 krb5kdc[22317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:41 dist-test-slave-df53 krb5kdc[22317](info): set up 2 sockets
Sep 27 22:56:41 dist-test-slave-df53 krb5kdc[22317](info): commencing operation
krb5kdc: starting...
W20260927 22:56:42.013877 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.008s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:42 dist-test-slave-df53 krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:56:42.256251 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45911
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:42.578274 22330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:42.578557 22330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:42.578682 22330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:42.592585 22330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:42.592744 22330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:42.592864 22330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:42.592965 22330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:42.593039 22330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:42.606418 22330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:42.607988 22330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:42.609256 22330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:42.621986 22336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:42.622033 22335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:42.622169 22338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:42.622752 22330 server_base.cc:1034] running on GCE node
I20260927 22:56:42.623442 22330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:42.624711 22330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:42.625927 22330 hybrid_clock.cc:648] HybridClock initialized: now 1790549802625871 us; error 46 us; skew 500 ppm
I20260927 22:56:42.626240 22330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:42 dist-test-slave-df53 krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:42.629098 22330 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:56:42.630446 22330 webserver.cc:460] Webserver started at http://127.0.97.62:33555/ using document root <none> and password file <none>
I20260927 22:56:42.631067 22330 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:42.631206 22330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:42.631543 22330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:42.634363 22330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "b6ae72b2304c428b8a571686f2fd3f6c"
format_stamp: "Formatted at 2026-09-27 22:56:42 on dist-test-slave-df53"
server_key: "e32bbf8bc5956d148968b6186eeb8e63"
server_key_iv: "0f7fe1103d9f96f20f9c36a2fd27c0fa"
server_key_version: "encryptionkey@0"
I20260927 22:56:42.635099 22330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "b6ae72b2304c428b8a571686f2fd3f6c"
format_stamp: "Formatted at 2026-09-27 22:56:42 on dist-test-slave-df53"
server_key: "e32bbf8bc5956d148968b6186eeb8e63"
server_key_iv: "0f7fe1103d9f96f20f9c36a2fd27c0fa"
server_key_version: "encryptionkey@0"
I20260927 22:56:42.639875 22330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 22:56:42.643117 22345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:42.644312 22330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:56:42.644490 22330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "b6ae72b2304c428b8a571686f2fd3f6c"
format_stamp: "Formatted at 2026-09-27 22:56:42 on dist-test-slave-df53"
server_key: "e32bbf8bc5956d148968b6186eeb8e63"
server_key_iv: "0f7fe1103d9f96f20f9c36a2fd27c0fa"
server_key_version: "encryptionkey@0"
I20260927 22:56:42.644704 22330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:42.650318 22330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:42.654224 22330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:42.654497 22330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:42.672384 22330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45911
I20260927 22:56:42.672432 22396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45911 every 8 connection(s)
I20260927 22:56:42.673635 22330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:42.676370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 22330
I20260927 22:56:42.676559 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:42.676975 388 external_mini_cluster.cc:1428] Setting key c90195a1efbf473ea3429c3244c1a449
I20260927 22:56:42.677312 22397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:42.686251 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:42.678645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46331 (local address 127.0.97.62:45911)
0927 22:56:42.679499 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:42.679515 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:42.679566 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:56:42.682256 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:42.682285 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:42.682733 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:42.683196 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:42.684204 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:42.684219 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:42.684237 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:42.684261 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:42.684334 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:42.684363 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:42.684614 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:42.684745 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:56:42.684966 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":344,"threads_started":1}
I20260927 22:56:42.689707 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:42.691772 22397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c: Bootstrap starting.
I20260927 22:56:42.695040 22397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:42.696148 22397 log.cc:824] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:42.696386 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:42.693869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39197 (local address 127.0.97.62:45911)
0927 22:56:42.694053 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:42.694060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:42.694395 (+ 335us) server_negotiation.cc:407] Connection header received
0927 22:56:42.694582 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:42.694587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:42.694657 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:42.694779 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:42.695667 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:42.695672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:42.695677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:42.695693 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:42.695739 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:42.695754 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:42.695872 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:42.695991 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 22:56:42.696055 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:56:42.698959 22397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c: No bootstrap required, opened a new log
I20260927 22:56:42.706533 22397 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae72b2304c428b8a571686f2fd3f6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45911 } }
I20260927 22:56:42.706892 22397 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:42.707010 22397 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ae72b2304c428b8a571686f2fd3f6c, State: Initialized, Role: FOLLOWER
I20260927 22:56:42.707567 22397 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae72b2304c428b8a571686f2fd3f6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45911 } }
I20260927 22:56:42.707782 22397 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:42.707928 22397 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:42.708091 22397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:42.710382 22397 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae72b2304c428b8a571686f2fd3f6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45911 } }
I20260927 22:56:42.710811 22397 leader_election.cc:302] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ae72b2304c428b8a571686f2fd3f6c; no voters:
I20260927 22:56:42.711282 22397 leader_election.cc:288] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:42.711408 22407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:42.712184 22407 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [term 1 LEADER]: Becoming Leader. State: Replica: b6ae72b2304c428b8a571686f2fd3f6c, State: Running, Role: LEADER
I20260927 22:56:42.712687 22407 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae72b2304c428b8a571686f2fd3f6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45911 } }
I20260927 22:56:42.713404 22397 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:42.715557 22409 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ae72b2304c428b8a571686f2fd3f6c. Latest consensus state: current_term: 1 leader_uuid: "b6ae72b2304c428b8a571686f2fd3f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae72b2304c428b8a571686f2fd3f6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45911 } } }
I20260927 22:56:42.716043 22409 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:42.716253 22408 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6ae72b2304c428b8a571686f2fd3f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ae72b2304c428b8a571686f2fd3f6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45911 } } }
I20260927 22:56:42.716696 22408 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:42.719105 22413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:42.723973 22413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:42.732385 22413 catalog_manager.cc:1347] Generated new cluster ID: c2ebbbb7b707447ca0c6220d5bb50a96
I20260927 22:56:42.732570 22413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:42.748291 22413 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:42.748653 22413 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:42.757526 22413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6ae72b2304c428b8a571686f2fd3f6c: Generated new TSK 0
I20260927 22:56:42.758482 22413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:42.773555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22330
W20260927 22:56:42.774854 22392 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46331
W20260927 22:56:42.775556 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22330: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:42.775712 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 22330
2026-09-27T22:56:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1846 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:42 dist-test-slave-df53 krb5kdc[22428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:42 dist-test-slave-df53 krb5kdc[22428](info): set up 2 sockets
Sep 27 22:56:42 dist-test-slave-df53 krb5kdc[22428](info): commencing operation
krb5kdc: starting...
W20260927 22:56:43.885959 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:44 dist-test-slave-df53 krb5kdc[22428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:56:44.120579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39109
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:44.432837 22441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:44.433099 22441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:44.433259 22441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:44.440228 22441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:44.440347 22441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:44.440454 22441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:44.440557 22441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:44.440645 22441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:44.450028 22441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:44.451202 22441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:44.452430 22441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:44.463562 22446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:44.463783 22447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:44.463819 22449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:44.464823 22441 server_base.cc:1034] running on GCE node
I20260927 22:56:44.465646 22441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:44.466926 22441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:44.468158 22441 hybrid_clock.cc:648] HybridClock initialized: now 1790549804468126 us; error 40 us; skew 500 ppm
I20260927 22:56:44.468497 22441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:44 dist-test-slave-df53 krb5kdc[22428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:44.471711 22441 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:56:44.473088 22441 webserver.cc:460] Webserver started at http://127.0.97.62:40661/ using document root <none> and password file <none>
I20260927 22:56:44.473788 22441 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:44.473927 22441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:44.474223 22441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:44.477007 22441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "84ac53cccf3547d2b9fe3485c57794aa"
format_stamp: "Formatted at 2026-09-27 22:56:44 on dist-test-slave-df53"
server_key: "1e3d3fab6549815ce49f8c2394c5b701"
server_key_iv: "ded3af166997524e9bdc151124c424d7"
server_key_version: "encryptionkey@0"
I20260927 22:56:44.477815 22441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "84ac53cccf3547d2b9fe3485c57794aa"
format_stamp: "Formatted at 2026-09-27 22:56:44 on dist-test-slave-df53"
server_key: "1e3d3fab6549815ce49f8c2394c5b701"
server_key_iv: "ded3af166997524e9bdc151124c424d7"
server_key_version: "encryptionkey@0"
I20260927 22:56:44.482699 22441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:56:44.486038 22456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:44.487284 22441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:56:44.487463 22441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "84ac53cccf3547d2b9fe3485c57794aa"
format_stamp: "Formatted at 2026-09-27 22:56:44 on dist-test-slave-df53"
server_key: "1e3d3fab6549815ce49f8c2394c5b701"
server_key_iv: "ded3af166997524e9bdc151124c424d7"
server_key_version: "encryptionkey@0"
I20260927 22:56:44.487663 22441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:44.506161 22441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:44.510586 22441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:44.510855 22441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:44.528609 22441 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39109
I20260927 22:56:44.528640 22507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39109 every 8 connection(s)
I20260927 22:56:44.530319 22441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:44.531085 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 22441
I20260927 22:56:44.531286 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:44.531746 388 external_mini_cluster.cc:1428] Setting key 341715814f63ab76ceb5a609beef9d2b
I20260927 22:56:44.534809 22508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:56:44 dist-test-slave-df53 krb5kdc[22428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:44.547128 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa: Bootstrap starting.
I20260927 22:56:44.550375 22508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:44.551347 22508 log.cc:824] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:44.554148 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa: No bootstrap required, opened a new log
I20260927 22:56:44.554589 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.533482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47941 (local address 127.0.97.62:39109)
0927 22:56:44.534274 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.534289 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.534339 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:56:44.536577 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.536604 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.537084 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.537494 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:44.538821 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.541006 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.542299 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.543302 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.543475 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.547467 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:44.547474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:44.547487 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:44.547523 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:44.550693 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.551353 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.551359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.551365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.551591 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.551962 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.551967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.551974 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.552455 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:44.552752 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.553193 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.553577 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":321,"threads_started":1}
I20260927 22:56:44.558331 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:44.562094 22508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ac53cccf3547d2b9fe3485c57794aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39109 } }
I20260927 22:56:44.562512 22508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:44.562651 22508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ac53cccf3547d2b9fe3485c57794aa, State: Initialized, Role: FOLLOWER
I20260927 22:56:44.563239 22508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ac53cccf3547d2b9fe3485c57794aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39109 } }
I20260927 22:56:44.563515 22508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:44.563665 22508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:44.563856 22508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:44.565938 22508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ac53cccf3547d2b9fe3485c57794aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39109 } }
I20260927 22:56:44.566380 22508 leader_election.cc:302] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84ac53cccf3547d2b9fe3485c57794aa; no voters:
I20260927 22:56:44.566905 22508 leader_election.cc:288] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:44.567027 22518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:44.568763 22508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:44.573840 22518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [term 1 LEADER]: Becoming Leader. State: Replica: 84ac53cccf3547d2b9fe3485c57794aa, State: Running, Role: LEADER
I20260927 22:56:44.574616 22518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ac53cccf3547d2b9fe3485c57794aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39109 } }
I20260927 22:56:44.577714 22519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84ac53cccf3547d2b9fe3485c57794aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ac53cccf3547d2b9fe3485c57794aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39109 } } }
I20260927 22:56:44.578130 22519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:44.580581 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.562508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56497 (local address 127.0.97.62:39109)
0927 22:56:44.562707 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.562716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.563213 (+ 497us) server_negotiation.cc:407] Connection header received
0927 22:56:44.563414 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.563419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.563515 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.563610 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:44.565014 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.565786 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.567393 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.568280 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.568397 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.572333 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:44.572342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:44.572350 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:44.572398 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:44.576403 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.577199 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.577204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.577207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.577407 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.577872 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.577877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.577880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.578247 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:44.578679 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.579808 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.580086 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 22:56:44.584120 22520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84ac53cccf3547d2b9fe3485c57794aa. Latest consensus state: current_term: 1 leader_uuid: "84ac53cccf3547d2b9fe3485c57794aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ac53cccf3547d2b9fe3485c57794aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39109 } } }
I20260927 22:56:44.584360 22520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa [sys.catalog]: This master's current role is: LEADER
W20260927 22:56:44.589960 22532 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:56:44.590092 22532 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:56:44.590271 22533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:44.591988 22533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:44.599659 22533 catalog_manager.cc:1347] Generated new cluster ID: b1d04772cdae41748707db05b441b3df
I20260927 22:56:44.599764 22533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:44.627458 22533 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:44.629269 22533 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:44.656533 22533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84ac53cccf3547d2b9fe3485c57794aa: Generated new TSK 0
I20260927 22:56:44.657542 22533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:44.665109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22441
I20260927 22:56:44.666184 22503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:44.758324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 22441
2026-09-27T22:56:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:44 dist-test-slave-df53 krb5kdc[22540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:44 dist-test-slave-df53 krb5kdc[22540](info): set up 2 sockets
Sep 27 22:56:44 dist-test-slave-df53 krb5kdc[22540](info): commencing operation
krb5kdc: starting...
W20260927 22:56:45.853962 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:46 dist-test-slave-df53 krb5kdc[22540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:56:46.094901 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46879
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:46.415831 22553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:46.416145 22553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:46.416330 22553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:46.423619 22553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:46.423766 22553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:46.423866 22553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:46.423975 22553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:46.424046 22553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:46.433874 22553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:46.435122 22553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:46.436372 22553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:46.448076 22558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:46.448051 22561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:46.448033 22559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:46.449430 22553 server_base.cc:1034] running on GCE node
I20260927 22:56:46.450320 22553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:46.451790 22553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:46.452993 22553 hybrid_clock.cc:648] HybridClock initialized: now 1790549806452933 us; error 51 us; skew 500 ppm
I20260927 22:56:46.453352 22553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:46 dist-test-slave-df53 krb5kdc[22540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:46.456596 22553 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:56:46.458155 22553 webserver.cc:460] Webserver started at http://127.0.97.62:37891/ using document root <none> and password file <none>
I20260927 22:56:46.458776 22553 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:46.458875 22553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:46.459124 22553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:46.462036 22553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "623c849734dd4839801ca1f75f6c7eb4"
format_stamp: "Formatted at 2026-09-27 22:56:46 on dist-test-slave-df53"
server_key: "bdf1a229f3e5927bceb10d6e42b2c4df"
server_key_iv: "5f0c80be651025613155339eef8beade"
server_key_version: "encryptionkey@0"
I20260927 22:56:46.462728 22553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "623c849734dd4839801ca1f75f6c7eb4"
format_stamp: "Formatted at 2026-09-27 22:56:46 on dist-test-slave-df53"
server_key: "bdf1a229f3e5927bceb10d6e42b2c4df"
server_key_iv: "5f0c80be651025613155339eef8beade"
server_key_version: "encryptionkey@0"
I20260927 22:56:46.467733 22553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 22:56:46.471119 22568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:46.472481 22553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:56:46.472662 22553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "623c849734dd4839801ca1f75f6c7eb4"
format_stamp: "Formatted at 2026-09-27 22:56:46 on dist-test-slave-df53"
server_key: "bdf1a229f3e5927bceb10d6e42b2c4df"
server_key_iv: "5f0c80be651025613155339eef8beade"
server_key_version: "encryptionkey@0"
I20260927 22:56:46.472846 22553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:46.488302 22553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:46.492661 22553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:46.492937 22553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:46.511726 22553 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46879
I20260927 22:56:46.511754 22619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46879 every 8 connection(s)
I20260927 22:56:46.513123 22553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:46.517210 22620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:46.517810 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 22553
I20260927 22:56:46.517992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:46.518539 388 external_mini_cluster.cc:1428] Setting key 97db8803d9cfb851e49b27446898eef5
I20260927 22:56:46.530586 22620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4: Bootstrap starting.
I20260927 22:56:46.532131 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:46.520282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50553 (local address 127.0.97.62:46879)
0927 22:56:46.521083 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:46.521093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:46.521135 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:56:46.523367 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:46.523389 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:46.523783 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:46.524116 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:46.524698 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:46.526099 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:46.527953 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:46.528625 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:46.528745 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:46.530001 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:46.530008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:46.530029 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:46.530538 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:46.530614 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:46.530636 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:46.530770 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:46.530970 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 22:56:46.531216 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":391,"threads_started":1}
I20260927 22:56:46.533918 22620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:46.535188 22620 log.cc:824] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:46.535970 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:46.539356 22620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4: No bootstrap required, opened a new log
I20260927 22:56:46.546788 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:46.540722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60105 (local address 127.0.97.62:46879)
0927 22:56:46.540869 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:46.540873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:46.541464 (+ 591us) server_negotiation.cc:407] Connection header received
0927 22:56:46.541691 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:46.541696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:46.541769 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:46.541888 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:46.542371 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:46.543115 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:46.544391 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:46.544863 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:46.545023 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:46.546095 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:46.546103 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:46.546107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:46.546123 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:46.546160 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:46.546174 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:46.546280 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:46.546420 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 22:56:46.546506 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:56:46.547394 22620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623c849734dd4839801ca1f75f6c7eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46879 } }
I20260927 22:56:46.547816 22620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:46.547930 22620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623c849734dd4839801ca1f75f6c7eb4, State: Initialized, Role: FOLLOWER
I20260927 22:56:46.548488 22620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623c849734dd4839801ca1f75f6c7eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46879 } }
I20260927 22:56:46.548688 22620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:46.548781 22620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:46.548894 22620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:46.551090 22620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623c849734dd4839801ca1f75f6c7eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46879 } }
I20260927 22:56:46.551594 22620 leader_election.cc:302] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 623c849734dd4839801ca1f75f6c7eb4; no voters:
I20260927 22:56:46.552287 22620 leader_election.cc:288] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:46.552389 22630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:46.553117 22630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [term 1 LEADER]: Becoming Leader. State: Replica: 623c849734dd4839801ca1f75f6c7eb4, State: Running, Role: LEADER
I20260927 22:56:46.553723 22630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623c849734dd4839801ca1f75f6c7eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46879 } }
I20260927 22:56:46.554435 22620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:46.556298 22632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 623c849734dd4839801ca1f75f6c7eb4. Latest consensus state: current_term: 1 leader_uuid: "623c849734dd4839801ca1f75f6c7eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623c849734dd4839801ca1f75f6c7eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46879 } } }
I20260927 22:56:46.556818 22632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:46.557682 22638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:46.557596 22631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "623c849734dd4839801ca1f75f6c7eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "623c849734dd4839801ca1f75f6c7eb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46879 } } }
I20260927 22:56:46.559142 22631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:46.562834 22638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:46.571815 22638 catalog_manager.cc:1347] Generated new cluster ID: 9f84e883ae3c44108038be4d0e487c63
I20260927 22:56:46.571955 22638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:46.581735 22638 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:46.583005 22638 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:46.603395 22638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 623c849734dd4839801ca1f75f6c7eb4: Generated new TSK 0
I20260927 22:56:46.604272 22638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:46.613760 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22553
W20260927 22:56:46.615129 22615 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50553
W20260927 22:56:46.615787 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:46.615934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 22553
2026-09-27T22:56:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:46 dist-test-slave-df53 krb5kdc[22651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:46 dist-test-slave-df53 krb5kdc[22651](info): set up 2 sockets
Sep 27 22:56:46 dist-test-slave-df53 krb5kdc[22651](info): commencing operation
krb5kdc: starting...
W20260927 22:56:47.706120 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.009s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:47 dist-test-slave-df53 krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:56:47.944824 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45461
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:48.265053 22664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:48.265318 22664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:48.265522 22664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:48.272572 22664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:48.272667 22664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:48.272801 22664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:48.272912 22664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:48.272993 22664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:48.282678 22664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:48.283869 22664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:48.285123 22664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:48.296680 22669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:48.296694 22670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:48.296680 22672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:48.297569 22664 server_base.cc:1034] running on GCE node
I20260927 22:56:48.298410 22664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:48.299723 22664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:48.300925 22664 hybrid_clock.cc:648] HybridClock initialized: now 1790549808300890 us; error 35 us; skew 500 ppm
I20260927 22:56:48.301262 22664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:48 dist-test-slave-df53 krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:48.311687 22664 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:56:48.313264 22664 webserver.cc:460] Webserver started at http://127.0.97.62:44211/ using document root <none> and password file <none>
I20260927 22:56:48.314015 22664 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:48.314154 22664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:48.314455 22664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:48.317514 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "db63a8055baa40fdba077b5b88ea0e73"
format_stamp: "Formatted at 2026-09-27 22:56:48 on dist-test-slave-df53"
server_key: "260e8eee492cf390e4efdefebfa28977"
server_key_iv: "f2a5cc6818ea4322949b3eb2217ab207"
server_key_version: "encryptionkey@0"
I20260927 22:56:48.318315 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "db63a8055baa40fdba077b5b88ea0e73"
format_stamp: "Formatted at 2026-09-27 22:56:48 on dist-test-slave-df53"
server_key: "260e8eee492cf390e4efdefebfa28977"
server_key_iv: "f2a5cc6818ea4322949b3eb2217ab207"
server_key_version: "encryptionkey@0"
I20260927 22:56:48.323238 22664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:56:48.326522 22679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:48.327709 22664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:48.327886 22664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "db63a8055baa40fdba077b5b88ea0e73"
format_stamp: "Formatted at 2026-09-27 22:56:48 on dist-test-slave-df53"
server_key: "260e8eee492cf390e4efdefebfa28977"
server_key_iv: "f2a5cc6818ea4322949b3eb2217ab207"
server_key_version: "encryptionkey@0"
I20260927 22:56:48.328073 22664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:48.333554 22664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:48.337695 22664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:48.337971 22664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:48.355765 22664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45461
I20260927 22:56:48.355811 22730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45461 every 8 connection(s)
I20260927 22:56:48.357105 22664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:48.360546 22731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:48.366802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 22664
I20260927 22:56:48.366983 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:48.367393 388 external_mini_cluster.cc:1428] Setting key 0c24a4c46306d9bacec5f4d49588a35d
I20260927 22:56:48.370492 22731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73: Bootstrap starting.
I20260927 22:56:48.373875 22731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:48.374604 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:48.368994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39495 (local address 127.0.97.62:45461)
0927 22:56:48.369747 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:48.369758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:48.369795 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:56:48.371411 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:48.371432 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:48.371745 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:48.372063 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:48.373072 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:48.373079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:48.373091 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:48.373104 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:48.373158 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:48.373176 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:48.373307 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:48.373470 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 22:56:48.373681 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":338,"threads_started":1}
I20260927 22:56:48.374923 22731 log.cc:824] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:48.377969 22731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73: No bootstrap required, opened a new log
I20260927 22:56:48.378448 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:48.385918 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:48.383585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60115 (local address 127.0.97.62:45461)
0927 22:56:48.383741 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:48.383745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:48.384037 (+ 292us) server_negotiation.cc:407] Connection header received
0927 22:56:48.384194 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:48.384197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:48.384243 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:48.384318 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:48.385226 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:48.385230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:48.385233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:56:48.385246 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:48.385281 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:56:48.385294 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:48.385389 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:48.385520 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:56:48.385566 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 22:56:48.388197 22731 raft_consensus.cc:348] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db63a8055baa40fdba077b5b88ea0e73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45461 } }
I20260927 22:56:48.388556 22731 raft_consensus.cc:374] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:48.388655 22731 raft_consensus.cc:729] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db63a8055baa40fdba077b5b88ea0e73, State: Initialized, Role: FOLLOWER
I20260927 22:56:48.389315 22731 consensus_queue.cc:260] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db63a8055baa40fdba077b5b88ea0e73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45461 } }
I20260927 22:56:48.389520 22731 raft_consensus.cc:388] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:48.389684 22731 raft_consensus.cc:482] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:48.389859 22731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:48.391968 22731 raft_consensus.cc:504] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db63a8055baa40fdba077b5b88ea0e73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45461 } }
I20260927 22:56:48.392455 22731 leader_election.cc:302] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db63a8055baa40fdba077b5b88ea0e73; no voters:
I20260927 22:56:48.393008 22731 leader_election.cc:288] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:48.393116 22741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:48.393824 22741 raft_consensus.cc:686] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [term 1 LEADER]: Becoming Leader. State: Replica: db63a8055baa40fdba077b5b88ea0e73, State: Running, Role: LEADER
I20260927 22:56:48.394366 22741 consensus_queue.cc:237] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db63a8055baa40fdba077b5b88ea0e73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45461 } }
I20260927 22:56:48.395056 22731 sys_catalog.cc:564] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:48.396863 22743 sys_catalog.cc:455] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db63a8055baa40fdba077b5b88ea0e73. Latest consensus state: current_term: 1 leader_uuid: "db63a8055baa40fdba077b5b88ea0e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db63a8055baa40fdba077b5b88ea0e73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45461 } } }
I20260927 22:56:48.397234 22743 sys_catalog.cc:458] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:48.398066 22742 sys_catalog.cc:455] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db63a8055baa40fdba077b5b88ea0e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db63a8055baa40fdba077b5b88ea0e73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45461 } } }
I20260927 22:56:48.398486 22742 sys_catalog.cc:458] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:48.398934 22746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:48.404162 22746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:48.412681 22746 catalog_manager.cc:1347] Generated new cluster ID: c244a41ad55a417fa3ee80d2dffd4546
I20260927 22:56:48.412789 22746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:48.428933 22746 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:48.429327 22746 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:48.442324 22746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db63a8055baa40fdba077b5b88ea0e73: Generated new TSK 0
I20260927 22:56:48.443095 22746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:48.452373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22664
W20260927 22:56:48.453455 22726 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39495
W20260927 22:56:48.454221 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22664: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:56:48.454335 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 22664
2026-09-27T22:56:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1838 ms)
[----------] 18 tests from AuthTokenIssuingTest (33956 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:48 dist-test-slave-df53 krb5kdc[22763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:48 dist-test-slave-df53 krb5kdc[22763](info): set up 2 sockets
Sep 27 22:56:48 dist-test-slave-df53 krb5kdc[22763](info): commencing operation
krb5kdc: starting...
W20260927 22:56:49.554482 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:49 dist-test-slave-df53 krb5kdc[22763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:56:49.797098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41625
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41625,127.0.97.61:40505,127.0.97.60:46083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:50.114821 22776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:56:50.115147 22776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:50.115262 22776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:50.115413 22776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:50.122664 22776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:50.122742 22776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:50.122853 22776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:50.122936 22776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:50.123026 22776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:50.132364 22776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41625,127.0.97.61:40505,127.0.97.60:46083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:50.133615 22776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:50.134900 22776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:50.146469 22781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:50.146395 22782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:50.146348 22784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:50.147492 22776 server_base.cc:1034] running on GCE node
I20260927 22:56:50.148203 22776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:50.149505 22776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:50.150758 22776 hybrid_clock.cc:648] HybridClock initialized: now 1790549810150704 us; error 33 us; skew 500 ppm
I20260927 22:56:50.151086 22776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:50 dist-test-slave-df53 krb5kdc[22763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:50.154213 22776 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:56:50.155637 22776 webserver.cc:460] Webserver started at http://127.0.97.62:44765/ using document root <none> and password file <none>
I20260927 22:56:50.156284 22776 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:50.156414 22776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:50.156719 22776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:50.159596 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "f745d9a8437f4250b38acd01914143fd"
format_stamp: "Formatted at 2026-09-27 22:56:50 on dist-test-slave-df53"
server_key: "f4632929ed0a42e74c208889abc11410"
server_key_iv: "885ed6b86641633996c660a6560e464e"
server_key_version: "encryptionkey@0"
I20260927 22:56:50.160571 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "f745d9a8437f4250b38acd01914143fd"
format_stamp: "Formatted at 2026-09-27 22:56:50 on dist-test-slave-df53"
server_key: "f4632929ed0a42e74c208889abc11410"
server_key_iv: "885ed6b86641633996c660a6560e464e"
server_key_version: "encryptionkey@0"
I20260927 22:56:50.165364 22776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260927 22:56:50.168649 22791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:50.169930 22776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:50.170096 22776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "f745d9a8437f4250b38acd01914143fd"
format_stamp: "Formatted at 2026-09-27 22:56:50 on dist-test-slave-df53"
server_key: "f4632929ed0a42e74c208889abc11410"
server_key_iv: "885ed6b86641633996c660a6560e464e"
server_key_version: "encryptionkey@0"
I20260927 22:56:50.170297 22776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:50.207139 22776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:50.211723 22776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:50.212004 22776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:50.230424 22776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41625
I20260927 22:56:50.230450 22842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41625 every 8 connection(s)
I20260927 22:56:50.231779 22776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:50.235309 22843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:50.240600 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 22776
I20260927 22:56:50.240810 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:50.241550 388 external_mini_cluster.cc:1428] Setting key de490303c72068cd660aa2a381eb3e3a
I20260927 22:56:50.242388 22843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:50 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260927 22:56:50.265995 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.246137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41625 (local address 127.0.0.1:35182)
0927 22:56:50.247289 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.247341 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:50.249988 (+ 2647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:50.250882 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:50.250892 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:50.251706 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:50.252044 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:50.252061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.253674 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.253681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:50.254727 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:50.254738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.255629 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.255636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:50.255807 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:50.256452 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:50.256482 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:50.258808 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:50.261928 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:50.261940 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:50.261951 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:50.262419 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:50.262810 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:50.262817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:50.262820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:50.262981 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:50.263683 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:50.263692 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:50.263929 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 22:56:50.264161 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 22:56:50.264723 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":140,"spinlock_wait_cycles":18176,"thread_start_us":661,"threads_started":1}
I20260927 22:56:50.266028 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.246149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.0.97.62:41625)
0927 22:56:50.247017 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:50.247030 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:50.247480 (+ 450us) server_negotiation.cc:407] Connection header received
0927 22:56:50.250393 (+ 2913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:50.250399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:50.250616 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:50.250796 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:50.252265 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.253495 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.254871 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.255400 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.255605 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:50.259031 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:50.259039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:50.259050 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:50.259081 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:50.261795 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:50.262544 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:50.262555 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:50.262558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:50.262702 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:50.263093 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:50.263098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:50.263101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:50.263547 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:50.263750 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:50.264288 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 22:56:50.264587 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":349,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 22:56:50.268944 22843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:56:50.269951 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.269575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55798)
0927 22:56:50.269755 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.269850 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 22:56:50.270179 22792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
I20260927 22:56:50.271659 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.271312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55804)
0927 22:56:50.271471 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.271539 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:56:50.271881 22843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40505: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 22:56:50.296679 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40505
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41625,127.0.97.61:40505,127.0.97.60:46083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:50.321286 22843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } attempt: 1
I20260927 22:56:50.322538 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.322050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55816)
0927 22:56:50.322241 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.322344 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:56:50.323510 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.323160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55832)
0927 22:56:50.323308 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.323382 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:56:50.323683 22843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40505: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
I20260927 22:56:50.391916 22843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } attempt: 2
I20260927 22:56:50.393020 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.392550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55834)
0927 22:56:50.392798 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.392888 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:56:50.394062 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.393727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55844)
0927 22:56:50.393883 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.393942 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:56:50.394232 22843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40505: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
I20260927 22:56:50.485492 22843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } attempt: 3
I20260927 22:56:50.486769 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.486305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55858)
0927 22:56:50.486535 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.486627 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:56:50.487866 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.487491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55874)
0927 22:56:50.487653 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.487738 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 22:56:50.488072 22843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40505: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
W20260927 22:56:50.615681 22850 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:56:50.616060 22850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:50.616197 22850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:50.616326 22850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:50.623464 22850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:50.623570 22850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:50.623698 22850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:50.623780 22850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:50.623906 22850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:50.631291 22843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } attempt: 4
I20260927 22:56:50.632421 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.631914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55880)
0927 22:56:50.632216 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.632300 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260927 22:56:50.633392 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.633080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55892)
0927 22:56:50.633215 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.633290 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:56:50.633499 22850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41625,127.0.97.61:40505,127.0.97.60:46083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
W20260927 22:56:50.633579 22843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40505: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40505: connect: Connection refused (error 111)
I20260927 22:56:50.634733 22850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:50.635946 22850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:50.647060 22856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:50.647105 22858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:50.647128 22855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:50.647957 22850 server_base.cc:1034] running on GCE node
I20260927 22:56:50.648738 22850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:50.650063 22850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:50.651353 22850 hybrid_clock.cc:648] HybridClock initialized: now 1790549810651283 us; error 54 us; skew 500 ppm
I20260927 22:56:50.651762 22850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:50 dist-test-slave-df53 krb5kdc[22763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:50.654881 22850 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 22:56:50.656123 22850 webserver.cc:460] Webserver started at http://127.0.97.61:40609/ using document root <none> and password file <none>
I20260927 22:56:50.656728 22850 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:50.656807 22850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:50.657042 22850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:50.659915 22850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/data/instance:
uuid: "d5bb55ab628f41f4b52f59d233750cd6"
format_stamp: "Formatted at 2026-09-27 22:56:50 on dist-test-slave-df53"
server_key: "39c7fd7d8efa8fe4353d7e9dbe599f5c"
server_key_iv: "79c7d2fa6952904d3025b2fc2e3fcf4e"
server_key_version: "encryptionkey@0"
I20260927 22:56:50.660609 22850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/wal/instance:
uuid: "d5bb55ab628f41f4b52f59d233750cd6"
format_stamp: "Formatted at 2026-09-27 22:56:50 on dist-test-slave-df53"
server_key: "39c7fd7d8efa8fe4353d7e9dbe599f5c"
server_key_iv: "79c7d2fa6952904d3025b2fc2e3fcf4e"
server_key_version: "encryptionkey@0"
I20260927 22:56:50.665226 22850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 22:56:50.668506 22865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:50.669771 22850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:50.669919 22850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/wal
uuid: "d5bb55ab628f41f4b52f59d233750cd6"
format_stamp: "Formatted at 2026-09-27 22:56:50 on dist-test-slave-df53"
server_key: "39c7fd7d8efa8fe4353d7e9dbe599f5c"
server_key_iv: "79c7d2fa6952904d3025b2fc2e3fcf4e"
server_key_version: "encryptionkey@0"
I20260927 22:56:50.670075 22850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:50.709832 22850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:50.713992 22850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:50.714285 22850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:50.731560 22850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40505
I20260927 22:56:50.731582 22916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40505 every 8 connection(s)
I20260927 22:56:50.732816 22850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:56:50.736203 22917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:50.738596 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 22850
I20260927 22:56:50.738778 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-1/wal/instance
I20260927 22:56:50.739279 388 external_mini_cluster.cc:1428] Setting key 13edd757a4d0a5ce1f1754b79473b576
I20260927 22:56:50.744779 22917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:50 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260927 22:56:50.766664 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.748052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.0.97.62:41625)
0927 22:56:50.748276 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:50.748284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:50.749384 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 22:56:50.751301 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:50.751305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:50.751390 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:50.751522 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:50.753507 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.754229 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.756584 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.756975 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.757066 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:50.761018 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:50.761023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:50.761026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:50.761057 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:50.763667 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:50.764389 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:50.764402 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:50.764406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:50.764531 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:50.764933 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:50.764939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:50.764942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:50.765203 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:50.765354 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:50.766129 (+ 775us) server_negotiation.cc:299] Negotiation successful
0927 22:56:50.766340 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:56:50.767616 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.748430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41625 (local address 127.0.0.1:35196)
0927 22:56:50.749205 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.749255 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:50.751049 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:50.751776 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:50.751789 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:50.752544 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:50.753304 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:50.753330 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.754387 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.754395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:50.756444 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:50.756461 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.757206 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.757216 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:50.757537 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:50.758271 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:50.758304 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:50.760808 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:50.763811 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:50.763820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:50.763835 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:50.764260 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:50.764641 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:50.764646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:50.764650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:50.764822 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:50.765330 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:50.765337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:50.765726 (+ 389us) client_negotiation.cc:769] Sending connection context
0927 22:56:50.766024 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 22:56:50.766494 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":365,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260927 22:56:50.769339 22917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:50 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:56:50.783944 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.770178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55902)
0927 22:56:50.770386 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.770405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:50.770512 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:50.771475 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:50.771479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:50.771900 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:50.772024 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:50.772031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.773559 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.773566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:50.774578 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:50.774589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.775379 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.775385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:50.775565 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:50.776075 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:50.776102 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:50.778638 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:50.781632 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:50.781639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:50.781645 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:50.781981 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:50.782339 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:50.782344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:50.782348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:50.782486 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:50.783055 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:50.783060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:50.783185 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 22:56:50.783322 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 22:56:50.783572 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 22:56:50.784046 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.770366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.0.97.61:40505)
0927 22:56:50.771073 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:50.771078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:50.771102 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:56:50.771145 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:50.771151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:50.771322 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:50.771492 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:50.772194 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.773409 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.774699 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.775223 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.775363 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:50.778836 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:50.778847 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:50.778858 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:50.778883 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:50.781470 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:50.782112 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:50.782127 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:50.782130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:50.782232 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:50.782592 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:50.782597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:50.782600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:50.782903 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:50.783113 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:50.783468 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 22:56:50.783677 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":410,"threads_started":1}
I20260927 22:56:50.786664 22917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:56:50.787737 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.787313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57332)
0927 22:56:50.787484 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.787608 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:56:50.788026 22869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
I20260927 22:56:50.789450 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.789168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57336)
0927 22:56:50.789288 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.789358 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 22:56:50.789775 22917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 22:56:50.798842 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46083
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41625,127.0.97.61:40505,127.0.97.60:46083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:50.839079 22917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } attempt: 1
I20260927 22:56:50.840231 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.839803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57340)
0927 22:56:50.839994 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.840084 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:56:50.841440 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.840979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57354)
0927 22:56:50.841165 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.841293 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:56:50.841741 22917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
I20260927 22:56:50.909993 22917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } attempt: 2
I20260927 22:56:50.911103 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.910642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57370)
0927 22:56:50.910833 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.910939 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:56:50.912134 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.911796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57384)
0927 22:56:50.911943 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.912022 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:56:50.912389 22917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
I20260927 22:56:50.932852 22843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } attempt: 5
Sep 27 22:56:50 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:56:50.947289 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.933540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55914)
0927 22:56:50.933772 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.933793 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:50.933930 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:50.934319 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:50.934322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:50.934700 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:50.934868 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:50.934876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.936423 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.936427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:50.937276 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:50.937287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.938271 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.938276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:50.938406 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:50.939004 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:50.939021 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:50.941315 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:50.945122 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:50.945126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:50.945129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:50.945466 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:50.945888 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:50.945892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:50.945894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:50.946039 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:50.946650 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:50.946653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:50.946773 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 22:56:50.946876 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 22:56:50.947055 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 22:56:50.947496 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.933673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.0.97.61:40505)
0927 22:56:50.933905 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:50.933912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:50.933935 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:56:50.934037 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:50.934045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:50.934158 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:50.934280 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:50.935045 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.936266 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.937429 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:50.938132 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:50.938254 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:50.941532 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:50.941541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:50.941547 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:50.941620 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:50.944994 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:50.945609 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:50.945615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:50.945619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:50.945776 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:50.946153 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:50.946160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:50.946165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:50.946532 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:50.946687 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:50.947001 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:56:50.947215 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:56:50.948474 22843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:56:50.949432 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.949094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57392)
0927 22:56:50.949239 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.949308 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 22:56:50.950313 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:50.950013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57394)
0927 22:56:50.950165 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:50.950229 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 22:56:50.950495 22843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
I20260927 22:56:51.001719 22843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } attempt: 1
I20260927 22:56:51.002839 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.002372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57398)
0927 22:56:51.002579 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.002678 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:56:51.003633 22917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } attempt: 3
I20260927 22:56:51.003762 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.003448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57404)
0927 22:56:51.003587 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.003657 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:56:51.004056 22843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
I20260927 22:56:51.004591 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.004186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57408)
0927 22:56:51.004375 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.004451 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:56:51.005576 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.005173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57424)
0927 22:56:51.005388 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.005465 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260927 22:56:51.005823 22917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
I20260927 22:56:51.072293 22843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } attempt: 2
I20260927 22:56:51.073565 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.073004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57426)
0927 22:56:51.073256 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.073376 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 22:56:51.074607 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.074230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57430)
0927 22:56:51.074385 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.074471 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:56:51.074813 22843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
W20260927 22:56:51.119086 22924 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:56:51.119517 22924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:51.119644 22924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:51.119791 22924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:51.126919 22924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:51.126995 22924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:51.127091 22924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:51.127179 22924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:51.127261 22924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:51.136535 22924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41625,127.0.97.61:40505,127.0.97.60:46083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:51.137758 22924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:51.138983 22924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:56:51.149111 22917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } attempt: 4
W20260927 22:56:51.149798 22929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:51.150331 22932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:51.149856 22930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:51.150815 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.150226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57434)
0927 22:56:51.150542 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.150666 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 22:56:51.151304 22924 server_base.cc:1034] running on GCE node
I20260927 22:56:51.151793 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.151480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57444)
0927 22:56:51.151628 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.151692 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:56:51.152012 22917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
I20260927 22:56:51.152197 22924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:51.153420 22924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:51.154654 22924 hybrid_clock.cc:648] HybridClock initialized: now 1790549811154573 us; error 57 us; skew 500 ppm
I20260927 22:56:51.154973 22924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:51 dist-test-slave-df53 krb5kdc[22763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:51.158051 22924 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 22:56:51.159351 22924 webserver.cc:460] Webserver started at http://127.0.97.60:33663/ using document root <none> and password file <none>
I20260927 22:56:51.160071 22924 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:51.160197 22924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:51.160496 22924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:51.163314 22924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/data/instance:
uuid: "dff54e4917714e85bcfb68bc67f7c3ee"
format_stamp: "Formatted at 2026-09-27 22:56:51 on dist-test-slave-df53"
server_key: "cba9edd8ae0a371556cc43387f3a191b"
server_key_iv: "eec5c410055882d3c75f23c4a8c7a9ca"
server_key_version: "encryptionkey@0"
I20260927 22:56:51.164069 22924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/wal/instance:
uuid: "dff54e4917714e85bcfb68bc67f7c3ee"
format_stamp: "Formatted at 2026-09-27 22:56:51 on dist-test-slave-df53"
server_key: "cba9edd8ae0a371556cc43387f3a191b"
server_key_iv: "eec5c410055882d3c75f23c4a8c7a9ca"
server_key_version: "encryptionkey@0"
I20260927 22:56:51.168787 22924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:56:51.172268 22939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:51.173636 22924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:56:51.173802 22924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/wal
uuid: "dff54e4917714e85bcfb68bc67f7c3ee"
format_stamp: "Formatted at 2026-09-27 22:56:51 on dist-test-slave-df53"
server_key: "cba9edd8ae0a371556cc43387f3a191b"
server_key_iv: "eec5c410055882d3c75f23c4a8c7a9ca"
server_key_version: "encryptionkey@0"
I20260927 22:56:51.174060 22924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:51.181010 22843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } attempt: 3
I20260927 22:56:51.182034 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.181556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57452)
0927 22:56:51.181808 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.181887 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 22:56:51.182914 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.182601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57468)
0927 22:56:51.182762 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.182821 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 22:56:51.183126 22843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46083: connect: Connection refused (error 111)
I20260927 22:56:51.190330 22924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:51.194170 22924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:51.194425 22924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:51.211845 22924 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46083
I20260927 22:56:51.211869 22990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46083 every 8 connection(s)
I20260927 22:56:51.213125 22924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:56:51.216679 22991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:51.220816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 22924
I20260927 22:56:51.221006 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549465302610-388-0/minicluster-data/master-2/wal/instance
I20260927 22:56:51.221426 388 external_mini_cluster.cc:1428] Setting key e183c7f284201d3f7ce6691255103331
I20260927 22:56:51.223559 22991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:51 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 22:56:51 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:56:51.244095 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.225994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41625 (local address 127.0.0.1:35208)
0927 22:56:51.226750 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.226793 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:51.226982 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:51.227404 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:51.227408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:51.227960 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:51.228495 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:51.228513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.229483 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.229489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:51.230730 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:51.230742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.231639 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.231645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:51.231908 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.232650 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:51.232676 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:51.235418 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:51.239519 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:51.239530 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:51.239541 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:51.240252 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:51.240691 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:51.240697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:51.240704 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:51.240923 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:51.241539 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:51.241547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:51.242029 (+ 482us) client_negotiation.cc:769] Sending connection context
0927 22:56:51.242334 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 22:56:51.242821 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":2,"spinlock_wait_cycles":1408,"thread_start_us":372,"threads_started":1}
I20260927 22:56:51.244114 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.223129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.0.97.60:46083)
0927 22:56:51.223899 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.223910 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.223950 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:56:51.225707 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.225727 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.226222 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.226681 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.228182 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.229645 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.231496 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.232099 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.232235 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.236837 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.236852 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.236875 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.236915 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.240494 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.241255 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.241263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.241267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.241412 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.241867 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.241871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.241874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.242293 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.242574 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.243103 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.243275 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":546,"spinlock_wait_cycles":26240,"thread_start_us":342,"threads_started":1}
I20260927 22:56:51.245026 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.225985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.0.97.62:41625)
0927 22:56:51.226228 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.226236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.226877 (+ 641us) server_negotiation.cc:407] Connection header received
0927 22:56:51.227144 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.227152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.227241 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.227357 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.228660 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.229330 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.230871 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.231383 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.231540 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.235624 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.235629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.235632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.235691 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.239371 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.240378 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.240384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.240389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.240575 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.241042 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.241045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.241047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.241377 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.241565 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.244543 (+ 2978us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.244777 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 22:56:51.246457 22991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:51 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 22:56:51 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:51.260705 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.247124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40505 (local address 127.0.0.1:55920)
0927 22:56:51.247289 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.247308 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:51.247401 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:51.247715 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:51.247719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:51.248128 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:51.248260 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:51.248268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.249569 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.249574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:51.250752 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:51.250763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.251675 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.251681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:51.251858 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.252392 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:51.252415 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:51.254560 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:51.258193 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:51.258196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:51.258199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:51.258539 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:51.259059 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:51.259062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:51.259065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:51.259205 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:51.259818 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:51.259826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:51.259986 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 22:56:51.260171 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 22:56:51.260442 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:56:51.261155 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.247234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.0.97.61:40505)
0927 22:56:51.247393 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.247397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.247413 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:56:51.247513 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.247517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.247572 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.247652 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.248412 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.249425 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.250882 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.251526 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.251672 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.254743 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.254752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.254758 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.254802 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.258043 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.258668 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.258675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.258680 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.258876 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.259320 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.259325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.259329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.259673 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.259822 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.260292 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.260551 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:56:51.262183 22991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:56:51.262406 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.247065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.0.97.62:41625)
0927 22:56:51.247237 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.247242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.247676 (+ 434us) server_negotiation.cc:407] Connection header received
0927 22:56:51.247724 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.247728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.247795 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.247894 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.249222 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.250268 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.251714 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.252319 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.252444 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.256323 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.256330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.256335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.256386 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.259597 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.260387 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.260392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.260395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.260578 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.260966 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.260973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.260976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.261348 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.261563 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.261956 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.262162 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 22:56:51 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:56:51 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:56:51.275452 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.262905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57490)
0927 22:56:51.263096 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.263114 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:51.263212 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:51.263528 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:51.263532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:51.264032 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:51.264205 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:51.264217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.265296 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.265303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:51.266388 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:51.266399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.267150 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.267157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:51.267368 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.267890 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:51.267912 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:51.269970 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:51.272823 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:51.272828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:51.272832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:51.273252 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:51.273822 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:51.273827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:51.273837 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:51.274009 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:51.274640 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:51.274647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:51.274825 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 22:56:51.274974 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:56:51.275205 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:56:51.275967 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.263057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.0.97.60:46083)
0927 22:56:51.263175 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.263179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.263193 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:56:51.263332 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.263336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.263402 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.263485 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.264358 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.265163 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.266521 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.267009 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.267106 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.270142 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.270146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.270149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.270175 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.272675 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.273456 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.273464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.273470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.273687 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.274127 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.274132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.274136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.274525 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.274677 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.275078 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.275393 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":28}
I20260927 22:56:51.278869 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.264044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.0.97.61:40505)
0927 22:56:51.264187 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.264191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.264205 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:56:51.264366 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.264370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.264480 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.264578 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.265883 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.266576 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.268073 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.268450 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.268568 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.272834 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.272840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.272844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.272884 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.276287 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.277006 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.277011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.277015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.277165 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.277573 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.277576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.277578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.277871 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.278079 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.278451 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.278619 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:56:51.281754 22991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee: Bootstrap starting.
I20260927 22:56:51.283481 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:51.285315 22991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:51.286571 22991 log.cc:824] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:51.289238 22991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee: No bootstrap required, opened a new log
I20260927 22:56:51.286803 22818 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f745d9a8437f4250b38acd01914143fd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35216
I20260927 22:56:51.296375 22991 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } }
I20260927 22:56:51.296702 22991 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff54e4917714e85bcfb68bc67f7c3ee, State: Initialized, Role: FOLLOWER
I20260927 22:56:51.297134 22991 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } }
I20260927 22:56:51.298062 22997 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } } }
I20260927 22:56:51.298424 22997 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:51.299108 22991 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:56:51.308955 23008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:56:51.309198 23008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:56:51.309228 22924 master_runner.cc:186] Error getting master registration for 127.0.97.62:41625: OK, instance_id { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" instance_seqno: 1790549810225169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:56:51.320114 22924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:56:51.360325 22843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } attempt: 4
Sep 27 22:56:51 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:56:51.376515 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.361113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57498)
0927 22:56:51.361330 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.361349 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:51.361486 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:51.361953 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:51.361957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:51.362477 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:51.362683 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:51.362694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.364140 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.364145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:51.365574 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:51.365616 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.366620 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.366626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:51.366812 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.367452 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:51.367469 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:51.369669 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:51.373581 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:51.373608 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:51.373615 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:51.374098 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:51.374516 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:51.374522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:51.374526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:51.374701 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:51.375508 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:51.375514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:51.375673 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 22:56:51.375823 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 22:56:51.376116 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:56:51.376562 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.361345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.0.97.60:46083)
0927 22:56:51.361562 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.361571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.361616 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:56:51.361699 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.361706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.361788 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.361921 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.362853 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.363988 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.365727 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.366451 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.366570 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.369866 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.369873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.369878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.369925 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.373456 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.374213 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.374218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.374222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.374362 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.374893 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.374901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.374905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.375371 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.375524 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.375967 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.376194 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:56:51.381165 22843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd: Bootstrap starting.
I20260927 22:56:51.384320 22843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:51.385349 22843 log.cc:824] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:51.387964 22843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd: No bootstrap required, opened a new log
I20260927 22:56:51.392901 22843 raft_consensus.cc:348] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } }
I20260927 22:56:51.393251 22843 raft_consensus.cc:729] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f745d9a8437f4250b38acd01914143fd, State: Initialized, Role: FOLLOWER
I20260927 22:56:51.393718 22843 consensus_queue.cc:260] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } }
I20260927 22:56:51.394639 23011 sys_catalog.cc:455] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } } }
I20260927 22:56:51.395013 23011 sys_catalog.cc:458] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:51.395717 22843 sys_catalog.cc:564] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:51.395874 22818 raft_consensus.cc:482] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:56:51.396234 22818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:51.399964 22818 raft_consensus.cc:504] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } }
I20260927 22:56:51.403708 22966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f745d9a8437f4250b38acd01914143fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dff54e4917714e85bcfb68bc67f7c3ee"
I20260927 22:56:51.404163 22966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:51.407617 22966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f745d9a8437f4250b38acd01914143fd in term 1.
I20260927 22:56:51.408357 22795 leader_election.cc:302] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff54e4917714e85bcfb68bc67f7c3ee, f745d9a8437f4250b38acd01914143fd; no voters:
I20260927 22:56:51.408785 23011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:51.416020 22818 leader_election.cc:288] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [CANDIDATE]: Term 1 election: Requested vote from peers d5bb55ab628f41f4b52f59d233750cd6 (127.0.97.61:40505), dff54e4917714e85bcfb68bc67f7c3ee (127.0.97.60:46083)
I20260927 22:56:51.405890 22776 master_runner.cc:186] Error getting master registration for 127.0.97.61:40505: OK, instance_id { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" instance_seqno: 1790549810726814 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:56:51.421231 22776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:56:51.432703 23011 raft_consensus.cc:686] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [term 1 LEADER]: Becoming Leader. State: Replica: f745d9a8437f4250b38acd01914143fd, State: Running, Role: LEADER
I20260927 22:56:51.433393 23011 consensus_queue.cc:237] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } }
I20260927 22:56:51.443336 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f745d9a8437f4250b38acd01914143fd. Latest consensus state: current_term: 1 leader_uuid: "f745d9a8437f4250b38acd01914143fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } } }
I20260927 22:56:51.443878 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [sys.catalog]: This master's current role is: LEADER
W20260927 22:56:51.445523 23022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:56:51.445729 23022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 22:56:51.446384 22945 tablet.cc:2369] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:51.451200 22917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } attempt: 5
I20260927 22:56:51.459892 23032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 22:56:51 dist-test-slave-df53 krb5kdc[22763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:56:51.461894 23032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 22:56:51.464855 22797 tablet.cc:2369] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:51.468793 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.451746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46083 (local address 127.0.0.1:57506)
0927 22:56:51.451901 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:51.451920 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:51.452025 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:51.452500 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:51.452507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:51.453028 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:51.453246 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:51.453264 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.454691 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.454700 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:51.456625 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:51.456643 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.457625 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.457631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:51.457872 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.458611 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:51.458635 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:51.461424 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:51.465897 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:51.465905 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:51.465911 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:51.466414 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:51.466862 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:51.466867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:51.466871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:51.467062 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:51.467783 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:51.467789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:51.467958 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 22:56:51.468110 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 22:56:51.468396 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:56:51.469684 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.452008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.0.97.60:46083)
0927 22:56:51.452189 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.452196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.452212 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:56:51.452265 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.452270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.452360 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.452462 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.453412 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.454566 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.456763 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.457462 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.457573 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.461653 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.461662 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.461668 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.461711 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.465743 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.466561 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.466566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.466569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.466719 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.467199 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.467205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.467210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.467554 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.467712 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.469091 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.469315 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:56:51.476806 22917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6: Bootstrap starting.
I20260927 22:56:51.484611 22917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:51.484967 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.440470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.0.97.61:40505)
0927 22:56:51.440716 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.440723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.441635 (+ 912us) server_negotiation.cc:407] Connection header received
0927 22:56:51.442057 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.442064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.442158 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.442233 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.446075 (+ 3842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.447216 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.461313 (+ 14097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.462031 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.462153 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.476287 (+ 14134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.476293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.476298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.476335 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.480100 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.482277 (+ 2177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.482286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.482289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.482473 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.482948 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.482955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.482962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.483396 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.483652 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.484265 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.484554 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:56:51.486812 22917 log.cc:824] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:51.489254 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.441011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.0.97.62:41625)
0927 22:56:51.441259 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.441267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.462864 (+ 21597us) server_negotiation.cc:407] Connection header received
0927 22:56:51.463151 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.463156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.463235 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.463314 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.468513 (+ 5199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.469586 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.472235 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.472928 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.473035 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.477184 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.477193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.477198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.477241 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.481332 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.486415 (+ 5083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.486432 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.486435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.486633 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.487132 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.487140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.487146 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.487568 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.487847 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.488459 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.488785 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:56:51.494364 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.453004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.0.97.60:46083)
0927 22:56:51.458747 (+ 5743us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.458756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.467320 (+ 8564us) server_negotiation.cc:407] Connection header received
0927 22:56:51.467376 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.467381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.467468 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.467563 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.469612 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.470873 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.472863 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.473480 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.473649 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.487250 (+ 13601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:51.487257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:51.487262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:51.487304 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:51.490912 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.491796 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.491802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.491806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.491959 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:51.492398 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:51.492403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:51.492406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:51.492798 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:51.493031 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.493743 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.493977 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5560,"thread_start_us":5480,"threads_started":1}
I20260927 22:56:51.497047 22966 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [term 1 FOLLOWER]: Refusing update from remote peer f745d9a8437f4250b38acd01914143fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:51.498044 22917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6: No bootstrap required, opened a new log
I20260927 22:56:51.498003 23011 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:51.509344 22917 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } }
I20260927 22:56:51.509779 22917 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5bb55ab628f41f4b52f59d233750cd6, State: Initialized, Role: FOLLOWER
I20260927 22:56:51.510306 22917 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } }
I20260927 22:56:51.511703 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } } }
I20260927 22:56:51.512460 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:51.512843 22917 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:51.512941 22892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f745d9a8437f4250b38acd01914143fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d5bb55ab628f41f4b52f59d233750cd6"
I20260927 22:56:51.513383 22892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:51.517856 22892 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f745d9a8437f4250b38acd01914143fd in term 1.
I20260927 22:56:51.518986 22891 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [term 1 FOLLOWER]: Refusing update from remote peer f745d9a8437f4250b38acd01914143fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:51.520291 23011 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:51.524031 22997 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f745d9a8437f4250b38acd01914143fd. Latest consensus state: current_term: 1 leader_uuid: "f745d9a8437f4250b38acd01914143fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } } }
I20260927 22:56:51.524341 22997 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:51.528110 23035 mvcc.cc:205] Tried to move back new op lower bound from 7334092027874795520 to 7334092027637420032. Current Snapshot: MvccSnapshot[applied={T|T < 7334092027874795520}]
I20260927 22:56:51.529933 23037 mvcc.cc:205] Tried to move back new op lower bound from 7334092027874795520 to 7334092027637420032. Current Snapshot: MvccSnapshot[applied={T|T < 7334092027874795520}]
I20260927 22:56:51.544279 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f745d9a8437f4250b38acd01914143fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } } }
I20260927 22:56:51.544648 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:51.546844 23032 catalog_manager.cc:1347] Generated new cluster ID: fcab608a3491484faa5dcd434a16d486
I20260927 22:56:51.546957 23032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:51.547817 22997 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f745d9a8437f4250b38acd01914143fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } } }
I20260927 22:56:51.548059 22997 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:51.588624 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f745d9a8437f4250b38acd01914143fd. Latest consensus state: current_term: 1 leader_uuid: "f745d9a8437f4250b38acd01914143fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } } }
I20260927 22:56:51.589038 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:51.590317 23011 sys_catalog.cc:455] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f745d9a8437f4250b38acd01914143fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } } }
I20260927 22:56:51.590639 23011 sys_catalog.cc:458] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:51.600041 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f745d9a8437f4250b38acd01914143fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745d9a8437f4250b38acd01914143fd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } peers { permanent_uuid: "d5bb55ab628f41f4b52f59d233750cd6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40505 } } peers { permanent_uuid: "dff54e4917714e85bcfb68bc67f7c3ee" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46083 } } }
I20260927 22:56:51.600373 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:51.632923 23032 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:51.634954 23032 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:51.655741 23032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f745d9a8437f4250b38acd01914143fd: Generated new TSK 0
I20260927 22:56:51.656984 23032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:51.660189 23056 catalog_manager.cc:1259] Loaded cluster ID: fcab608a3491484faa5dcd434a16d486
I20260927 22:56:51.660279 23056 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6: loading cluster ID for follower catalog manager: success
I20260927 22:56:51.663967 23056 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6: acquiring CA information for follower catalog manager: success
I20260927 22:56:51.666318 23056 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d5bb55ab628f41f4b52f59d233750cd6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:56:51.786473 22921 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:51.772494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.0.97.61:40505)
0927 22:56:51.772682 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.772689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.784445 (+ 11756us) server_negotiation.cc:407] Connection header received
0927 22:56:51.784920 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.784927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.785034 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.785113 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.786213 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55932: BlockingRecv error: recv got EOF from 127.0.0.1:55932 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 22:56:51.788652 22845 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:51.780861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.0.97.62:41625)
0927 22:56:51.781083 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.781090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.787163 (+ 6073us) server_negotiation.cc:407] Connection header received
0927 22:56:51.787366 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.787372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.787504 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.787569 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.788421 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35240: BlockingRecv error: recv got EOF from 127.0.0.1:35240 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 22:56:51.795351 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:51.790057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.0.97.60:46083)
0927 22:56:51.790227 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.790234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.793660 (+ 3426us) server_negotiation.cc:407] Connection header received
0927 22:56:51.793897 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.793903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.793984 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.794053 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.795112 (+ 1059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57510: BlockingRecv error: recv got EOF from 127.0.0.1:57510 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:56:51.854152 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.833294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.0.97.61:40505)
0927 22:56:51.833480 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.833486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.846161 (+ 12675us) server_negotiation.cc:407] Connection header received
0927 22:56:51.846409 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.846416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.846663 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.846837 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:51.847490 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.848635 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.850785 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.851461 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.851565 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.852314 (+ 749us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:51.853105 (+ 791us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:51.853231 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.853698 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.853792 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:56:51.899833 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:51.890615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.0.97.60:46083)
0927 22:56:51.890787 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.890793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.897669 (+ 6876us) server_negotiation.cc:407] Connection header received
0927 22:56:51.898188 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.898195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.898317 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.898389 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.899565 (+ 1176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57522: BlockingRecv error: recv got EOF from 127.0.0.1:57522 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:56:51.938485 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.931880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.0.97.61:40505)
0927 22:56:51.932075 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.932081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.932102 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:56:51.932185 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.932191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.932293 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.932398 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:51.932902 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.934085 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.936094 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.936798 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.936888 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.937477 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:51.937660 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:51.937727 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.937974 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.938077 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 22:56:51.958751 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:51.956584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57524 (local address 127.0.97.60:46083)
0927 22:56:51.956809 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.956818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.957340 (+ 522us) server_negotiation.cc:407] Connection header received
0927 22:56:51.957557 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.957562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.957690 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.957791 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.958517 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57524: BlockingRecv error: recv got EOF from 127.0.0.1:57524 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:56:51.969059 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.963332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55958 (local address 127.0.97.61:40505)
0927 22:56:51.963553 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.963561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.964097 (+ 536us) server_negotiation.cc:407] Connection header received
0927 22:56:51.964285 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.964290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.964364 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.964459 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:51.964876 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.965707 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.967092 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.967550 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.967686 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.968219 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:51.968336 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:51.968429 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.968577 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.968670 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:56:51.981194 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:51.979105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.0.97.60:46083)
0927 22:56:51.979306 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.979314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.979839 (+ 525us) server_negotiation.cc:407] Connection header received
0927 22:56:51.980024 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.980029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.980110 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.980216 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:51.980960 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57532: BlockingRecv error: recv got EOF from 127.0.0.1:57532 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:56:51.993494 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:51.987889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.0.97.61:40505)
0927 22:56:51.988226 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:51.988235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:51.988719 (+ 484us) server_negotiation.cc:407] Connection header received
0927 22:56:51.988898 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:51.988904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:51.988989 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:51.989108 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:51.989519 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.990438 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.991739 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:51.992234 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:51.992370 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:51.992756 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:51.992860 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:51.992973 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:51.993116 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 22:56:51.993194 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
W20260927 22:56:52.005455 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:52.003373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.0.97.60:46083)
0927 22:56:52.003595 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.003601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.004168 (+ 567us) server_negotiation.cc:407] Connection header received
0927 22:56:52.004365 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.004370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.004458 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.004568 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:52.005265 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57540: BlockingRecv error: recv got EOF from 127.0.0.1:57540 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 22:56:52.024471 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:52.017438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.0.97.61:40505)
0927 22:56:52.017637 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.017643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.018222 (+ 579us) server_negotiation.cc:407] Connection header received
0927 22:56:52.018429 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.018434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.018520 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.018623 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:52.019122 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.020272 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.022276 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.023006 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.023132 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:52.023621 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:52.023798 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:52.023898 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:52.024083 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 22:56:52.024163 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 22:56:52.033761 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:52.031558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.0.97.60:46083)
0927 22:56:52.031747 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.031753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.032284 (+ 531us) server_negotiation.cc:407] Connection header received
0927 22:56:52.032433 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.032439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.032527 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.032638 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:52.033497 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57550: BlockingRecv error: recv got EOF from 127.0.0.1:57550 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:56:52.060014 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:52.052858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.0.97.61:40505)
0927 22:56:52.053061 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.053069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.053589 (+ 520us) server_negotiation.cc:407] Connection header received
0927 22:56:52.053801 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.053807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.053892 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.054015 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:52.054491 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.055695 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.057811 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.058524 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.058653 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:52.059191 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:52.059319 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:52.059413 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:52.059599 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 22:56:52.059702 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 22:56:52.068593 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:52.066477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.0.97.60:46083)
0927 22:56:52.066644 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.066651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.067148 (+ 497us) server_negotiation.cc:407] Connection header received
0927 22:56:52.067322 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.067327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.067421 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.067529 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:52.068388 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57556: BlockingRecv error: recv got EOF from 127.0.0.1:57556 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:56:52.118017 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:52.111631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.0.97.61:40505)
0927 22:56:52.111806 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.111813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.112342 (+ 529us) server_negotiation.cc:407] Connection header received
0927 22:56:52.112565 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.112573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.112701 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.112818 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:52.113308 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.114555 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.116080 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.116770 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.116934 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:52.117234 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:52.117369 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:52.117466 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:52.117571 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 22:56:52.117678 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:56:52.126012 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:52.123949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.0.97.60:46083)
0927 22:56:52.124191 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.124200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.124508 (+ 308us) server_negotiation.cc:407] Connection header received
0927 22:56:52.124679 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.124683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.124751 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.124830 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:52.125762 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57560: BlockingRecv error: recv got EOF from 127.0.0.1:57560 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:56:52.199450 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:52.193997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.0.97.61:40505)
0927 22:56:52.194216 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.194226 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.194695 (+ 469us) server_negotiation.cc:407] Connection header received
0927 22:56:52.194932 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.194936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.195003 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.195105 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:52.195598 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.196367 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.197731 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.198172 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.198311 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:52.198694 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:52.198806 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:52.198922 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:52.199085 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 22:56:52.199166 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 22:56:52.206426 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:52.204755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.0.97.60:46083)
0927 22:56:52.204974 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.204981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.205231 (+ 250us) server_negotiation.cc:407] Connection header received
0927 22:56:52.205353 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.205356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.205425 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.205512 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:52.206237 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57568: BlockingRecv error: recv got EOF from 127.0.0.1:57568 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:56:52.311259 23008 catalog_manager.cc:1259] Loaded cluster ID: fcab608a3491484faa5dcd434a16d486
I20260927 22:56:52.311403 23008 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee: loading cluster ID for follower catalog manager: success
I20260927 22:56:52.314203 23008 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee: acquiring CA information for follower catalog manager: success
I20260927 22:56:52.315969 23008 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dff54e4917714e85bcfb68bc67f7c3ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:56:52.342823 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:52.337803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.0.97.61:40505)
0927 22:56:52.338040 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.338047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.338342 (+ 295us) server_negotiation.cc:407] Connection header received
0927 22:56:52.338586 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.338590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.338653 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.338752 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:52.339164 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.339840 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.341133 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.341544 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.341736 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:52.342118 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:52.342228 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:52.342330 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:52.342489 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 22:56:52.342565 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:56:52.354112 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:52.348932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.0.97.60:46083)
0927 22:56:52.349113 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.349118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.349375 (+ 257us) server_negotiation.cc:407] Connection header received
0927 22:56:52.349515 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.349519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.349734 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.349833 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:52.350203 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.350866 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.352121 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.352515 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.352608 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:52.353038 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:52.353527 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:52.353743 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:52.353783 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 22:56:52.353845 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:56:52.368439 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:52.360432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.0.97.61:40505)
0927 22:56:52.360645 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.360654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.362201 (+ 1547us) server_negotiation.cc:407] Connection header received
0927 22:56:52.362490 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.362496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.362596 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.362659 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:52.363519 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.364443 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.366530 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.367046 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.367168 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:52.367661 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:52.367780 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:52.367845 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:52.368078 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:56:52.368157 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:56:52.368465 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:52.360299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.0.97.62:41625)
0927 22:56:52.361835 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.361842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.361876 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:56:52.362018 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.362024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.362489 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.362667 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:52.363069 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.364138 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.366259 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.366766 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.366860 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:52.367416 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:52.367824 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:52.367944 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:52.368081 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 22:56:52.368161 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1400,"thread_start_us":1302,"threads_started":1}
I20260927 22:56:52.368736 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:52.362333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.0.97.60:46083)
0927 22:56:52.362510 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:52.362516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:52.363155 (+ 639us) server_negotiation.cc:407] Connection header received
0927 22:56:52.363339 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:52.363344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:52.363413 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:52.363508 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:52.364024 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.364868 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.366947 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:52.367436 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:52.367556 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:52.368005 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:52.368132 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:52.368192 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:52.368414 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 22:56:52.368499 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:56:52.373767 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22776
I20260927 22:56:52.374866 22838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:52.485188 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 22776
I20260927 22:56:52.505625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22850
I20260927 22:56:52.506757 22912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:52.614850 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 22850
I20260927 22:56:52.634603 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 22924
I20260927 22:56:52.635620 22986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:52.742774 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 22924
2026-09-27T22:56:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4295 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:52 dist-test-slave-df53 krb5kdc[23169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:52 dist-test-slave-df53 krb5kdc[23169](info): set up 2 sockets
Sep 27 22:56:52 dist-test-slave-df53 krb5kdc[23169](info): commencing operation
krb5kdc: starting...
W20260927 22:56:53.858651 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/test-user.keytab.
Sep 27 22:56:54 dist-test-slave-df53 krb5kdc[23169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 22:56:54.116819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44665
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44665,127.0.97.61:46355,127.0.97.60:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:54.437130 23182 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:56:54.437484 23182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:54.437582 23182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:54.437747 23182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:54.444886 23182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:54.444967 23182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:54.445067 23182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:54.445154 23182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:54.445246 23182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:54.455077 23182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44665,127.0.97.61:46355,127.0.97.60:42155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:54.456529 23182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:54.457917 23182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:54.469497 23188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:54.469448 23190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:54.469535 23187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:54.470865 23182 server_base.cc:1034] running on GCE node
I20260927 22:56:54.471592 23182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:54.472889 23182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:54.474131 23182 hybrid_clock.cc:648] HybridClock initialized: now 1790549814474074 us; error 37 us; skew 500 ppm
I20260927 22:56:54.474452 23182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:54 dist-test-slave-df53 krb5kdc[23169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:54.477424 23182 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 22:56:54.478905 23182 webserver.cc:460] Webserver started at http://127.0.97.62:39835/ using document root <none> and password file <none>
I20260927 22:56:54.479532 23182 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:54.479616 23182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:54.479857 23182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:54.482705 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/data/instance:
uuid: "791b4f96163845cba1b9b98ca0f68b0c"
format_stamp: "Formatted at 2026-09-27 22:56:54 on dist-test-slave-df53"
server_key: "e1b26b78cf8abb2775533a7bc99f1af9"
server_key_iv: "07a0079ba1f876ff47f813d682d81ab6"
server_key_version: "encryptionkey@0"
I20260927 22:56:54.483420 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/wal/instance:
uuid: "791b4f96163845cba1b9b98ca0f68b0c"
format_stamp: "Formatted at 2026-09-27 22:56:54 on dist-test-slave-df53"
server_key: "e1b26b78cf8abb2775533a7bc99f1af9"
server_key_iv: "07a0079ba1f876ff47f813d682d81ab6"
server_key_version: "encryptionkey@0"
I20260927 22:56:54.488265 23182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:56:54.491539 23197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:54.492816 23182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:56:54.492988 23182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/wal
uuid: "791b4f96163845cba1b9b98ca0f68b0c"
format_stamp: "Formatted at 2026-09-27 22:56:54 on dist-test-slave-df53"
server_key: "e1b26b78cf8abb2775533a7bc99f1af9"
server_key_iv: "07a0079ba1f876ff47f813d682d81ab6"
server_key_version: "encryptionkey@0"
I20260927 22:56:54.493196 23182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:54.504880 23182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:54.508716 23182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:54.508981 23182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:54.527118 23182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44665
I20260927 22:56:54.527159 23248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44665 every 8 connection(s)
I20260927 22:56:54.528395 23182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:54.529793 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 23182
I20260927 22:56:54.529994 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-0/wal/instance
I20260927 22:56:54.530520 388 external_mini_cluster.cc:1428] Setting key cb984152e5a0910d5f791051e3b530d3
I20260927 22:56:54.532989 23249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:54.542299 23249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 27 22:56:54 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:54.568859 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.545268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.0.97.62:44665)
0927 22:56:54.546105 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:54.546125 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:54.546543 (+ 418us) server_negotiation.cc:407] Connection header received
0927 22:56:54.548505 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:54.548511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:54.548723 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:54.548936 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:54.550271 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:54.551442 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:54.552890 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:54.553423 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:54.553640 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:54.560173 (+ 6533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:54.560186 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:54.560204 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:54.560253 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:54.564078 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:54.564754 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:54.564766 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:54.564772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:54.564976 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:54.565351 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:54.565359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:54.565365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:54.566077 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:54.566291 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:54.567074 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 22:56:54.567561 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":595,"thread_start_us":387,"threads_started":1}
I20260927 22:56:54.569022 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.545269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44665 (local address 127.0.0.1:39350)
0927 22:56:54.546384 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.546420 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:54.548150 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:54.548938 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:54.548944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:54.549736 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:54.550060 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:54.550076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:54.551596 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:54.551601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:54.552745 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:54.552760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:54.553676 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:54.553682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:54.553858 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:54.554524 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:54.554554 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:54.559930 (+ 5376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:54.564210 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:54.564219 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:54.564228 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:54.564603 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:54.565086 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:54.565089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:54.565091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:54.565228 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:54.566232 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:54.566240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:54.566496 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 22:56:54.566782 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 22:56:54.567301 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":720,"threads_started":1}
I20260927 22:56:54.572121 23249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:56:54.573282 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.572911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52736)
0927 22:56:54.573078 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.573179 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 22:56:54.573578 23198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
I20260927 22:56:54.575217 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.574855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52740)
0927 22:56:54.575015 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.575087 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 22:56:54.575488 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46355: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 22:56:54.592499 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46355
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44665,127.0.97.61:46355,127.0.97.60:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:54.624797 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } attempt: 1
I20260927 22:56:54.625988 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.625547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52752)
0927 22:56:54.625768 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.625854 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 22:56:54.626968 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.626630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52758)
0927 22:56:54.626787 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.626854 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:56:54.627185 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46355: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
I20260927 22:56:54.695423 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } attempt: 2
I20260927 22:56:54.696732 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.696208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52764)
0927 22:56:54.696508 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.696599 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 22:56:54.697831 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.697402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52780)
0927 22:56:54.697548 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.697696 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:56:54.698020 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46355: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
I20260927 22:56:54.789361 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } attempt: 3
I20260927 22:56:54.790500 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.790076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52794)
0927 22:56:54.790268 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.790374 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 22:56:54.791628 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.791165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52800)
0927 22:56:54.791363 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.791461 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 22:56:54.791821 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46355: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
W20260927 22:56:54.905491 23256 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:56:54.905846 23256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:54.905970 23256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:54.906105 23256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:54.913048 23256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:54.913125 23256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:54.913239 23256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:54.913331 23256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:54.913417 23256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:54.923496 23256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44665,127.0.97.61:46355,127.0.97.60:42155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:46355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:54.924675 23256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:54.925978 23256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:56:54.935037 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } attempt: 4
I20260927 22:56:54.936493 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.935762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52804)
0927 22:56:54.936167 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.936276 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255}
W20260927 22:56:54.937495 23261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:54.937528 23262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:54.937848 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:54.937358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52818)
0927 22:56:54.937562 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:54.937696 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260927 22:56:54.938093 23264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:54.938174 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:46355: Network error: Client connection negotiation failed: client connection to 127.0.97.61:46355: connect: Connection refused (error 111)
I20260927 22:56:54.938690 23256 server_base.cc:1034] running on GCE node
I20260927 22:56:54.939368 23256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:54.940670 23256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:54.941900 23256 hybrid_clock.cc:648] HybridClock initialized: now 1790549814941840 us; error 39 us; skew 500 ppm
I20260927 22:56:54.942231 23256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:54 dist-test-slave-df53 krb5kdc[23169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:54.945183 23256 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 22:56:54.946528 23256 webserver.cc:460] Webserver started at http://127.0.97.61:44981/ using document root <none> and password file <none>
I20260927 22:56:54.947170 23256 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:54.947295 23256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:54.947598 23256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:54.950332 23256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/data/instance:
uuid: "d581a40effd94f34a3d050b7ff596a81"
format_stamp: "Formatted at 2026-09-27 22:56:54 on dist-test-slave-df53"
server_key: "8ad66b77742cae74ee92acfb9c135dd5"
server_key_iv: "1f1680bfbae1e0593a8144910d8b3e4e"
server_key_version: "encryptionkey@0"
I20260927 22:56:54.951054 23256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/wal/instance:
uuid: "d581a40effd94f34a3d050b7ff596a81"
format_stamp: "Formatted at 2026-09-27 22:56:54 on dist-test-slave-df53"
server_key: "8ad66b77742cae74ee92acfb9c135dd5"
server_key_iv: "1f1680bfbae1e0593a8144910d8b3e4e"
server_key_version: "encryptionkey@0"
I20260927 22:56:54.955775 23256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 22:56:54.958950 23271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:54.960151 23256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:56:54.960330 23256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/wal
uuid: "d581a40effd94f34a3d050b7ff596a81"
format_stamp: "Formatted at 2026-09-27 22:56:54 on dist-test-slave-df53"
server_key: "8ad66b77742cae74ee92acfb9c135dd5"
server_key_iv: "1f1680bfbae1e0593a8144910d8b3e4e"
server_key_version: "encryptionkey@0"
I20260927 22:56:54.960558 23256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:54.975521 23256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:54.979354 23256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:54.979628 23256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:54.997852 23256 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:46355
I20260927 22:56:54.997884 23322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:46355 every 8 connection(s)
I20260927 22:56:54.999171 23256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/data/info.pb
I20260927 22:56:55.002791 23323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:55.004444 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 23256
I20260927 22:56:55.004652 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-1/wal/instance
I20260927 22:56:55.005082 388 external_mini_cluster.cc:1428] Setting key a0fc415d5e06845ec4b886d1b63977ff
I20260927 22:56:55.009171 23323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260927 22:56:55.031972 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.011685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.97.62:44665)
0927 22:56:55.011919 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.011928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.013144 (+ 1216us) server_negotiation.cc:407] Connection header received
0927 22:56:55.016070 (+ 2926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.016074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.016142 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.016232 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.018287 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.019009 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.021413 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.021912 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.022018 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.026189 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.026194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.026197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.026231 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.028890 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.029631 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.029636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.029642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.029760 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.030181 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.030186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.030190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.030468 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.030625 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.031421 (+ 796us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.031639 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260927 22:56:55.032931 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.012155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44665 (local address 127.0.0.1:39358)
0927 22:56:55.012965 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.013018 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:55.015766 (+ 2748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:55.016504 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:55.016518 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:55.017297 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:55.018104 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.018130 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.019177 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.019186 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.021252 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.021269 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.022149 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.022159 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.022511 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.023228 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:55.023266 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:55.025972 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:55.029039 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.029048 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.029064 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.029488 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.029881 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.029886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.029889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.030060 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.030596 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:55.030604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:55.030933 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 22:56:55.031290 (+ 357us) client_negotiation.cc:240] Negotiation successful
0927 22:56:55.031797 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":318,"threads_started":1}
I20260927 22:56:55.034662 23323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:56:55.050333 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.035360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52830)
0927 22:56:55.035531 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.035552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:55.035646 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:55.036711 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:55.036716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:55.037130 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:55.037258 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.037265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.038699 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.038705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.039923 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.039938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.040826 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.040835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.041070 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.041740 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:55.041761 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:55.043663 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:55.048072 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.048076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.048079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.048420 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.048846 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.048849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.048851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.048981 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.049704 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:55.049707 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:55.049808 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 22:56:55.049917 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:56:55.050099 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:56:55.050874 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.035636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.0.97.61:46355)
0927 22:56:55.036325 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.036331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.036360 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:56:55.036398 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.036404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.036571 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.036727 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.037417 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.038550 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.040080 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.040643 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.040807 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.043913 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.043929 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.043950 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.044003 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.047933 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.048575 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.048586 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.048593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.048736 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.049088 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.049095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.049101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.049552 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.049787 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.050106 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.050384 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":398,"threads_started":1}
I20260927 22:56:55.053517 23323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:56:55.054744 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.054315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44164)
0927 22:56:55.054514 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.054621 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:56:55.055074 23275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
I20260927 22:56:55.056571 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.056272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44168)
0927 22:56:55.056402 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.056479 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:56:55.056825 23323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42155: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 22:56:55.067062 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
/tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42155
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44665,127.0.97.61:46355,127.0.97.60:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:55.106114 23323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } attempt: 1
I20260927 22:56:55.107234 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.106818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44178)
0927 22:56:55.107004 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.107091 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:56:55.108263 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.107939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44188)
0927 22:56:55.108086 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.108151 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:56:55.108507 23323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42155: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
I20260927 22:56:55.176743 23323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } attempt: 2
I20260927 22:56:55.177985 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.177523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44198)
0927 22:56:55.177745 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.177839 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 22:56:55.178999 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.178712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44212)
0927 22:56:55.178837 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.178898 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:56:55.179323 23323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42155: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
I20260927 22:56:55.237478 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } attempt: 5
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:56:55.251219 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.238387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.0.97.61:46355)
0927 22:56:55.238609 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.238614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.238636 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:56:55.238807 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.238812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.238901 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.239008 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.239807 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.240516 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.241729 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.242176 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.242311 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.245820 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.245825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.245828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.245859 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.248360 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.249299 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.249303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.249305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.249422 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.249903 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.249906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.249908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.250171 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.250292 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.250824 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.251008 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:56:55.251556 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.238251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52842)
0927 22:56:55.238441 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.238523 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:55.238658 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:55.239043 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:55.239048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:55.239471 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:55.239645 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.239653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.240661 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.240667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.241529 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.241541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.242321 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.242327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.242529 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.243143 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:55.243161 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:55.245640 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:55.248504 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.248511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.248518 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.249166 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.249550 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.249556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.249561 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.249786 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.250313 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:55.250321 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:55.250522 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 22:56:55.250726 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 22:56:55.251082 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:56:55.252674 23249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:56:55.253917 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.253512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44220)
0927 22:56:55.253715 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.253788 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 22:56:55.254786 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.254492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44222)
0927 22:56:55.254625 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.254704 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:56:55.254973 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42155: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
I20260927 22:56:55.270541 23323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } attempt: 3
I20260927 22:56:55.271639 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.271157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44234)
0927 22:56:55.271380 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.271481 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:56:55.272727 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.272291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44244)
0927 22:56:55.272539 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.272611 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260927 22:56:55.272958 23323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42155: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
I20260927 22:56:55.306180 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } attempt: 1
I20260927 22:56:55.307261 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.306796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44250)
0927 22:56:55.307006 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.307099 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 22:56:55.308280 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.307964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44264)
0927 22:56:55.308114 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.308182 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:56:55.308491 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42155: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
I20260927 22:56:55.376740 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } attempt: 2
I20260927 22:56:55.377900 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.377405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44268)
0927 22:56:55.377638 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.377749 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:56:55.378975 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.378594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44274)
0927 22:56:55.378748 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.378831 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:56:55.379252 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42155: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
W20260927 22:56:55.390163 23330 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:56:55.390496 23330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:55.390575 23330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:55.390728 23330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:55.398118 23330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:55.398193 23330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:55.398303 23330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:55.398389 23330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:55.398471 23330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:55.408051 23330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44665,127.0.97.61:46355,127.0.97.60:42155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 22:56:55.409281 23330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:55.410653 23330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:56:55.416165 23323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } attempt: 4
I20260927 22:56:55.417482 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.416875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44280)
0927 22:56:55.417221 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.417332 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 22:56:55.418514 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.418189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44296)
0927 22:56:55.418330 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.418414 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:56:55.418721 23323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42155: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
W20260927 22:56:55.422884 23338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:55.422855 23335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:55.423087 23336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:55.424149 23330 server_base.cc:1034] running on GCE node
I20260927 22:56:55.424966 23330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:55.426559 23330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:55.427819 23330 hybrid_clock.cc:648] HybridClock initialized: now 1790549815427758 us; error 52 us; skew 500 ppm
I20260927 22:56:55.428175 23330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:55.431926 23330 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 22:56:55.433357 23330 webserver.cc:460] Webserver started at http://127.0.97.60:33699/ using document root <none> and password file <none>
I20260927 22:56:55.434096 23330 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:55.434187 23330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:55.434458 23330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:55.437386 23330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/data/instance:
uuid: "e721c87df54a4421afe6e99a7f7e120b"
format_stamp: "Formatted at 2026-09-27 22:56:55 on dist-test-slave-df53"
server_key: "9b10e7806470dbe4b5228dec31deee24"
server_key_iv: "f2211c8b337adeb646683c922084a759"
server_key_version: "encryptionkey@0"
I20260927 22:56:55.438194 23330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/wal/instance:
uuid: "e721c87df54a4421afe6e99a7f7e120b"
format_stamp: "Formatted at 2026-09-27 22:56:55 on dist-test-slave-df53"
server_key: "9b10e7806470dbe4b5228dec31deee24"
server_key_iv: "f2211c8b337adeb646683c922084a759"
server_key_version: "encryptionkey@0"
I20260927 22:56:55.443171 23330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:56:55.446460 23345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:55.447837 23330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:56:55.447979 23330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/wal
uuid: "e721c87df54a4421afe6e99a7f7e120b"
format_stamp: "Formatted at 2026-09-27 22:56:55 on dist-test-slave-df53"
server_key: "9b10e7806470dbe4b5228dec31deee24"
server_key_iv: "f2211c8b337adeb646683c922084a759"
server_key_version: "encryptionkey@0"
I20260927 22:56:55.448210 23330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:55.470036 23330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:55.474264 23330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:55.474550 23330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:55.485496 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } attempt: 3
I20260927 22:56:55.486794 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.486243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44312)
0927 22:56:55.486522 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.486638 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 22:56:55.488047 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.487600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44314)
0927 22:56:55.487813 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.487910 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 22:56:55.488237 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42155: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42155: connect: Connection refused (error 111)
I20260927 22:56:55.493240 23330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42155
I20260927 22:56:55.493270 23396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42155 every 8 connection(s)
I20260927 22:56:55.494602 23330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/data/info.pb
I20260927 22:56:55.498270 23397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:55.499624 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu as pid 23330
I20260927 22:56:55.499804 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIqwSmZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549465302610-388-0/minicluster-data/master-2/wal/instance
I20260927 22:56:55.500373 388 external_mini_cluster.cc:1428] Setting key b13acdaa4e5af1ce9f08a7c61bf4c40e
I20260927 22:56:55.506835 23397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:55.523036 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.502121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.0.97.60:42155)
0927 22:56:55.502983 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.502995 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.503036 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:56:55.504560 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.504581 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.504955 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.505342 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.506849 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.508169 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.509588 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.510449 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.510626 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.515251 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.515262 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.515274 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.515316 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.518738 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.519406 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.519416 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.519427 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.519636 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.520000 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.520005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.520009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.520697 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.521001 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.521315 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.521763 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":431,"threads_started":1}
I20260927 22:56:55.524725 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.509129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.0.97.62:44665)
0927 22:56:55.509328 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.509336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.510082 (+ 746us) server_negotiation.cc:407] Connection header received
0927 22:56:55.510246 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.510252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.510332 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.510441 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.511873 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.512538 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.514292 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.514966 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.515102 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.518941 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.518952 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.518955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.518984 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.522117 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.522806 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.522811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.522814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.522978 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.523387 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.523392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.523394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.523742 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.523886 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.524283 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.524495 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:56:55.524850 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.509122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44665 (local address 127.0.0.1:39370)
0927 22:56:55.509927 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.509983 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:55.510128 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:55.510495 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:55.510499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:55.511370 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:55.511718 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.511731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.512701 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.512710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.514140 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.514154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.515251 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.515259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.515500 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.516116 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:55.516149 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:55.518771 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:55.522233 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.522242 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.522246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.522673 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.523087 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.523090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.523092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.523289 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.523856 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:55.523861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:55.523985 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 22:56:55.524106 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 22:56:55.524442 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":419,"threads_started":1}
I20260927 22:56:55.526477 23397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 22:56:55.541011 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.527077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:46355 (local address 127.0.0.1:52854)
0927 22:56:55.527314 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.527331 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:55.527414 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:55.527789 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:55.527794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:55.528261 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:55.528431 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.528441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.529735 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.529741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.530597 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.530621 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.531432 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.531438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.531630 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.532202 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:55.532223 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:55.534822 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:55.538655 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.538659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.538661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.538984 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.539423 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.539426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.539428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.539568 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.540069 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:55.540074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:55.540213 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 22:56:55.540347 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 22:56:55.540661 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260927 22:56:55.541039 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.527193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.0.97.61:46355)
0927 22:56:55.527347 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.527352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.527371 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:56:55.527540 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.527549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.527638 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.527749 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.528584 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.529557 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.530746 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.531278 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.531415 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.535004 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.535009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.535012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.535041 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.538535 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.539112 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.539119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.539123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.539308 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.539666 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.539672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.539676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.539938 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.540097 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.540442 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.540661 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 22:56:55.542037 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.526159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.0.97.62:44665)
0927 22:56:55.526303 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.526310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.526389 (+ 79us) server_negotiation.cc:407] Connection header received
0927 22:56:55.526571 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.526576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.526645 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.526728 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.528260 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.529333 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.531011 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.531685 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.531815 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.536375 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.536381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.536386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.536418 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.539596 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.540177 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.540184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.540187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.540299 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.540676 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.540682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.540685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.540940 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.541221 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.541479 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.541723 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:56:55.542394 23397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 22:56:55.556016 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.543275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.0.97.60:42155)
0927 22:56:55.543414 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.543421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.543440 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:56:55.543529 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.543533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.543598 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.543657 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.544553 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.545552 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.546991 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.547612 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.547740 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.551019 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.551023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.551026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.551051 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.553522 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.554325 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.554328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.554331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.554441 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.554853 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.554858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.554862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.555126 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.555230 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.555649 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.555792 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:56:55.556385 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.543145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44342)
0927 22:56:55.543296 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.543314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:55.543411 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:55.543825 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:55.543829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:55.544287 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:55.544400 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.544407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.545716 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.545722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.546865 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.546876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.547744 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.547747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.547974 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.548508 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:55.548531 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:55.550850 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:55.553714 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.553720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.553724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.554214 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.554556 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.554563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.554568 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.554724 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.555247 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:55.555253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:55.555416 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 22:56:55.555578 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 22:56:55.555997 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":98}
I20260927 22:56:55.558632 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.543672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.0.97.61:46355)
0927 22:56:55.543825 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.543831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.543844 (+ 13us) server_negotiation.cc:407] Connection header received
0927 22:56:55.543886 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.543891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.543964 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.544060 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.545360 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.546402 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.547873 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.548551 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.548681 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.553210 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.553215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.553218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.553245 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.556288 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.556957 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.556963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.556968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.557122 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.557450 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.557455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.557459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.557775 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.557965 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.558277 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.558411 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:56:55.562139 23397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b: Bootstrap starting.
I20260927 22:56:55.563455 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:55.566071 23397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:55.567075 23397 log.cc:824] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:55.566485 23224 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "791b4f96163845cba1b9b98ca0f68b0c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39384
I20260927 22:56:55.570144 23397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b: No bootstrap required, opened a new log
I20260927 22:56:55.581029 23397 raft_consensus.cc:348] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } }
I20260927 22:56:55.581379 23397 raft_consensus.cc:729] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e721c87df54a4421afe6e99a7f7e120b, State: Initialized, Role: FOLLOWER
I20260927 22:56:55.581892 23397 consensus_queue.cc:260] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } }
I20260927 22:56:55.582866 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } } }
I20260927 22:56:55.583256 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:55.583806 23397 sys_catalog.cc:564] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:56:55.593020 23414 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:56:55.593233 23414 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:56:55.591073 23330 master_runner.cc:186] Error getting master registration for 127.0.97.62:44665: OK, instance_id { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" instance_seqno: 1790549814521998 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:56:55.602033 23330 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:56:55.665498 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } attempt: 4
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:56:55.681561 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.666307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44352)
0927 22:56:55.666518 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.666541 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:55.666678 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:55.667051 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:55.667058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:55.667613 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:55.667837 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.667851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.669306 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.669311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.670797 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.670813 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.671829 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.671836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.672042 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.672717 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:55.672734 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:55.675070 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:55.679114 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.679119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.679123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.679609 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.680024 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.680029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.680032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.680203 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.680862 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:55.680865 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:55.680996 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 22:56:55.681107 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:56:55.681299 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 22:56:55.682046 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.666398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.0.97.60:42155)
0927 22:56:55.666632 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.666639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.666661 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:56:55.666802 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.666808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.666888 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.667003 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.667994 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.669152 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.670953 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.671678 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.671810 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.675257 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.675264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.675269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.675315 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.678967 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.679728 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.679734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.679738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.679896 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.680314 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.680319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.680323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.680747 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.680906 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.681249 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.681471 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:56:55.687718 23249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c: Bootstrap starting.
I20260927 22:56:55.691700 23249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:55.692934 23249 log.cc:824] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:55.695715 23249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c: No bootstrap required, opened a new log
I20260927 22:56:55.701064 23249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } }
I20260927 22:56:55.701375 23249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791b4f96163845cba1b9b98ca0f68b0c, State: Initialized, Role: FOLLOWER
I20260927 22:56:55.701896 23249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } }
I20260927 22:56:55.702852 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } } }
I20260927 22:56:55.703238 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:55.703943 23249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:55.704030 23224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:56:55.704213 23224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:55.708222 23224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } }
I20260927 22:56:55.712181 23372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "791b4f96163845cba1b9b98ca0f68b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e721c87df54a4421afe6e99a7f7e120b"
I20260927 22:56:55.712706 23372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:55.715663 23372 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791b4f96163845cba1b9b98ca0f68b0c in term 1.
W20260927 22:56:55.717343 23428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:56:55.717460 23428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:56:55.717901 23323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } attempt: 5
I20260927 22:56:55.717903 23201 leader_election.cc:302] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 791b4f96163845cba1b9b98ca0f68b0c, e721c87df54a4421afe6e99a7f7e120b; no voters:
I20260927 22:56:55.718346 23417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:55.724633 23224 leader_election.cc:288] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [CANDIDATE]: Term 1 election: Requested vote from peers d581a40effd94f34a3d050b7ff596a81 (127.0.97.61:46355), e721c87df54a4421afe6e99a7f7e120b (127.0.97.60:42155)
W20260927 22:56:55.727502 23351 tablet.cc:2369] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 22:56:55 dist-test-slave-df53 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 22:56:55.747535 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.718453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42155 (local address 127.0.0.1:44362)
0927 22:56:55.718706 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:55.718726 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:55.718836 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:55.719555 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:55.719560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:55.720057 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:55.720258 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.720270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.725975 (+ 5705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.725981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.727493 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:55.727509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.728749 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.728755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:55.728959 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.732759 (+ 3800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:55.732788 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:55.735702 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:55.739907 (+ 4205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.739913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.739917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.740422 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.745664 (+ 5242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:55.745669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:55.745673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:55.745837 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:55.746554 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:55.746559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:55.746720 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 22:56:55.746869 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:56:55.747149 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 22:56:55.748559 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.718602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.0.97.60:42155)
0927 22:56:55.718777 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.718783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.718801 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:56:55.719177 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.719182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.719263 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.719359 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.721692 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.722835 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.727717 (+ 4882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.728385 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.728541 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.735980 (+ 7439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.735987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.735991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.736034 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.739740 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.740582 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.740587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.740591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.740741 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.745980 (+ 5239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.745985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.745988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.746332 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.746491 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.747969 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.748192 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:56:55.754966 23323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81: Bootstrap starting.
I20260927 22:56:55.759071 23323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:55.760540 23323 log.cc:824] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81: Log is configured to *not* fsync() on all Append() calls
W20260927 22:56:55.761749 23203 tablet.cc:2369] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:55.723006 23182 master_runner.cc:186] Error getting master registration for 127.0.97.61:46355: OK, instance_id { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" instance_seqno: 1790549814992459 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:56:55.762521 23182 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:56:55.762884 23417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [term 1 LEADER]: Becoming Leader. State: Replica: 791b4f96163845cba1b9b98ca0f68b0c, State: Running, Role: LEADER
I20260927 22:56:55.763864 23417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } }
I20260927 22:56:55.767186 23323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81: No bootstrap required, opened a new log
I20260927 22:56:55.778321 23323 raft_consensus.cc:348] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } }
I20260927 22:56:55.778702 23323 raft_consensus.cc:729] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d581a40effd94f34a3d050b7ff596a81, State: Initialized, Role: FOLLOWER
I20260927 22:56:55.779232 23323 consensus_queue.cc:260] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } }
I20260927 22:56:55.781137 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791b4f96163845cba1b9b98ca0f68b0c. Latest consensus state: current_term: 1 leader_uuid: "791b4f96163845cba1b9b98ca0f68b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } } }
I20260927 22:56:55.781653 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:55.782873 23438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:55.790014 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } } }
I20260927 22:56:55.790426 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:55.792261 23323 sys_catalog.cc:564] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:55.793238 23298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "791b4f96163845cba1b9b98ca0f68b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d581a40effd94f34a3d050b7ff596a81"
I20260927 22:56:55.793738 23298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:55.797238 23438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:55.797485 23298 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791b4f96163845cba1b9b98ca0f68b0c in term 1.
I20260927 22:56:55.814635 23372 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [term 1 FOLLOWER]: Refusing update from remote peer 791b4f96163845cba1b9b98ca0f68b0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:55.815649 23430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:55.824931 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.786801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.0.97.61:46355)
0927 22:56:55.787010 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.787019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.787621 (+ 602us) server_negotiation.cc:407] Connection header received
0927 22:56:55.788441 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.788449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.788529 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.788600 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.794700 (+ 6100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.795823 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.798579 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.799224 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.799319 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.814526 (+ 15207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.814534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.814539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.814587 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.820889 (+ 6302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.822012 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.822018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.822022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.822180 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.822610 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.822616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.822619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.822975 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.823214 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.824163 (+ 949us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.824553 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:56:55.825508 23298 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [term 1 FOLLOWER]: Refusing update from remote peer 791b4f96163845cba1b9b98ca0f68b0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:55.826359 23430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:55.830744 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.785468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.0.97.62:44665)
0927 22:56:55.785660 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.785667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.786370 (+ 703us) server_negotiation.cc:407] Connection header received
0927 22:56:55.787341 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.787348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.787431 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.787521 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.791916 (+ 4395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.793040 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.800567 (+ 7527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.801436 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.801549 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.807851 (+ 6302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.807858 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.807862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.807911 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.811854 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.813337 (+ 1483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.813356 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.813360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.813540 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.826626 (+ 13086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.826632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.826635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.827041 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.827290 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.829707 (+ 2417us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.830040 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:56:55.846840 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:55.787860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.0.97.60:42155)
0927 22:56:55.788101 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:55.788108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:55.805346 (+ 17238us) server_negotiation.cc:407] Connection header received
0927 22:56:55.805647 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:55.805656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:55.805795 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:55.805890 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:55.814152 (+ 8262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.818302 (+ 4150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.834063 (+ 15761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:55.835110 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:55.835251 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:55.839604 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:55.839611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:55.839616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:55.839656 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:55.843252 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.844067 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.844072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.844093 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.844256 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:55.844691 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:55.844696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:55.844700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:55.845059 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:55.845293 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:55.845912 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 22:56:55.846252 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:56:55.848397 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791b4f96163845cba1b9b98ca0f68b0c. Latest consensus state: current_term: 1 leader_uuid: "791b4f96163845cba1b9b98ca0f68b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } } }
I20260927 22:56:55.848734 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:55.851092 23443 mvcc.cc:205] Tried to move back new op lower bound from 7334092045566353408 to 7334092045378465792. Current Snapshot: MvccSnapshot[applied={T|T < 7334092045566353408}]
W20260927 22:56:55.866173 23457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:56:55.866297 23457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:56:55.873823 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791b4f96163845cba1b9b98ca0f68b0c. Latest consensus state: current_term: 1 leader_uuid: "791b4f96163845cba1b9b98ca0f68b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } } }
I20260927 22:56:55.874142 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:55.881186 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "791b4f96163845cba1b9b98ca0f68b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } } }
I20260927 22:56:55.881456 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:55.892762 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "791b4f96163845cba1b9b98ca0f68b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } } }
I20260927 22:56:55.893023 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:55.908741 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "791b4f96163845cba1b9b98ca0f68b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } } }
I20260927 22:56:55.909020 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:55.914243 23438 catalog_manager.cc:1347] Generated new cluster ID: ba716c1b18b64ebd863b3071f91e7fe7
I20260927 22:56:55.914345 23438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:55.911193 23445 mvcc.cc:205] Tried to move back new op lower bound from 7334092045566353408 to 7334092045378465792. Current Snapshot: MvccSnapshot[applied={T|T < 7334092045566353408}]
I20260927 22:56:55.932003 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "791b4f96163845cba1b9b98ca0f68b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791b4f96163845cba1b9b98ca0f68b0c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44665 } } peers { permanent_uuid: "d581a40effd94f34a3d050b7ff596a81" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 46355 } } peers { permanent_uuid: "e721c87df54a4421afe6e99a7f7e120b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42155 } } }
I20260927 22:56:55.932222 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:55.945066 23438 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:55.947005 23438 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:55.968724 23438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 791b4f96163845cba1b9b98ca0f68b0c: Generated new TSK 0
I20260927 22:56:55.970260 23438 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:56:56.136616 23252 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.115104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.0.97.62:44665)
0927 22:56:56.115355 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.115363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.134169 (+ 18806us) server_negotiation.cc:407] Connection header received
0927 22:56:56.134510 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.134518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.134636 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.134724 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.136276 (+ 1552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39408: BlockingRecv error: recv got EOF from 127.0.0.1:39408 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 22:56:56.139264 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.121132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.0.97.61:46355)
0927 22:56:56.121336 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.121342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.137496 (+ 16154us) server_negotiation.cc:407] Connection header received
0927 22:56:56.137823 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.137832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.137943 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.138027 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.139021 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52892: BlockingRecv error: recv got EOF from 127.0.0.1:52892 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 22:56:56.141110 23400 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.134440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.0.97.60:42155)
0927 22:56:56.135013 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.135022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.138827 (+ 3805us) server_negotiation.cc:407] Connection header received
0927 22:56:56.139394 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.139399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.139493 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.139567 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.140811 (+ 1244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44386: BlockingRecv error: recv got EOF from 127.0.0.1:44386 (error 108)
Metrics: {"server-negotiator.queue_time_us":416}
W20260927 22:56:56.184158 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.171389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52896 (local address 127.0.97.61:46355)
0927 22:56:56.171551 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.171557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.182140 (+ 10583us) server_negotiation.cc:407] Connection header received
0927 22:56:56.182605 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.182611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.182749 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.182825 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.183822 (+ 997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52896: BlockingRecv error: recv got EOF from 127.0.0.1:52896 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 22:56:56.232127 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.219415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.0.97.61:46355)
0927 22:56:56.219592 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.219597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.230209 (+ 10612us) server_negotiation.cc:407] Connection header received
0927 22:56:56.230672 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.230682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.230782 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.230860 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.231893 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52904: BlockingRecv error: recv got EOF from 127.0.0.1:52904 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 22:56:56.251514 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.249262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.0.97.61:46355)
0927 22:56:56.249496 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.249506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.250023 (+ 517us) server_negotiation.cc:407] Connection header received
0927 22:56:56.250250 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.250262 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.250348 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.250472 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.251197 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52910: BlockingRecv error: recv got EOF from 127.0.0.1:52910 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 22:56:56.260036 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.258044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.0.97.61:46355)
0927 22:56:56.258248 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.258257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.258731 (+ 474us) server_negotiation.cc:407] Connection header received
0927 22:56:56.258918 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.258934 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.258995 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.259101 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.259831 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52916: BlockingRecv error: recv got EOF from 127.0.0.1:52916 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 22:56:56.272888 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.270885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.0.97.61:46355)
0927 22:56:56.271115 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.271124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.271618 (+ 494us) server_negotiation.cc:407] Connection header received
0927 22:56:56.271812 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.271819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.271885 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.271996 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.272647 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52918: BlockingRecv error: recv got EOF from 127.0.0.1:52918 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 22:56:56.295095 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.292766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.0.97.61:46355)
0927 22:56:56.293039 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.293049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.293629 (+ 580us) server_negotiation.cc:407] Connection header received
0927 22:56:56.293885 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.293889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.293962 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.294071 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.294843 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52932: BlockingRecv error: recv got EOF from 127.0.0.1:52932 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260927 22:56:56.337385 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.335364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.0.97.61:46355)
0927 22:56:56.335595 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.335602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.335872 (+ 270us) server_negotiation.cc:407] Connection header received
0927 22:56:56.336097 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.336103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.336189 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.336314 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.337088 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52946: BlockingRecv error: recv got EOF from 127.0.0.1:52946 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 22:56:56.406319 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.404532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.0.97.61:46355)
0927 22:56:56.404697 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.404701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.405123 (+ 422us) server_negotiation.cc:407] Connection header received
0927 22:56:56.405291 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.405295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.405367 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.405472 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.406150 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52950: BlockingRecv error: recv got EOF from 127.0.0.1:52950 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 22:56:56.539733 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.537761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.0.97.61:46355)
0927 22:56:56.537984 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.537990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.538292 (+ 302us) server_negotiation.cc:407] Connection header received
0927 22:56:56.538522 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.538526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.538597 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.538676 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.539532 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52952: BlockingRecv error: recv got EOF from 127.0.0.1:52952 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:56:56.595167 23414 catalog_manager.cc:1259] Loaded cluster ID: ba716c1b18b64ebd863b3071f91e7fe7
I20260927 22:56:56.595268 23414 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b: loading cluster ID for follower catalog manager: success
I20260927 22:56:56.597734 23414 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b: acquiring CA information for follower catalog manager: success
I20260927 22:56:56.599411 23414 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e721c87df54a4421afe6e99a7f7e120b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:56:56.800913 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:56:56.799160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.0.97.61:46355)
0927 22:56:56.799402 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:56.799408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:56.799646 (+ 238us) server_negotiation.cc:407] Connection header received
0927 22:56:56.799799 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:56.799803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:56.799878 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:56.799970 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:56.800677 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52964: BlockingRecv error: recv got EOF from 127.0.0.1:52964 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:56:56.868146 23457 catalog_manager.cc:1259] Loaded cluster ID: ba716c1b18b64ebd863b3071f91e7fe7
I20260927 22:56:56.868278 23457 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81: loading cluster ID for follower catalog manager: success
I20260927 22:56:56.870733 23457 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81: acquiring CA information for follower catalog manager: success
I20260927 22:56:56.872447 23457 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d581a40effd94f34a3d050b7ff596a81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:56:57.324365 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:57.318943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.0.97.61:46355)
0927 22:56:57.319457 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:57.319462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:57.319481 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:56:57.319657 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:57.319661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:57.319809 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:57.319916 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:57.320359 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:57.321083 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:57.322360 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:57.322757 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:57.322848 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:57.323273 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:57.323778 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:57.323892 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:57.323973 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 22:56:57.324035 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":264,"threads_started":1}
I20260927 22:56:57.335937 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:57.330194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.0.97.60:42155)
0927 22:56:57.330647 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:57.330654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:57.330926 (+ 272us) server_negotiation.cc:407] Connection header received
0927 22:56:57.331161 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:57.331165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:57.331308 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:57.331514 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:57.331881 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:57.332552 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:57.333883 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:57.334273 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:57.334417 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:57.334875 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:57.335349 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:57.335515 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:57.335568 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 22:56:57.335643 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":238,"threads_started":1}
I20260927 22:56:57.359055 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:57.341770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.0.97.62:44665)
0927 22:56:57.352567 (+ 10797us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:57.352575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:57.352595 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:56:57.352730 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:57.352736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:57.352962 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:57.353132 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:57.353790 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:57.354729 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:57.356654 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:57.357298 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:57.357450 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:57.357971 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:57.358393 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:57.358504 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:57.358618 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 22:56:57.358716 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10665,"thread_start_us":10555,"threads_started":1}
I20260927 22:56:57.359174 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:57.342196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.0.97.60:42155)
0927 22:56:57.348220 (+ 6024us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:57.348228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:57.352997 (+ 4769us) server_negotiation.cc:407] Connection header received
0927 22:56:57.353267 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:57.353275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:57.353352 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:57.353433 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:57.354242 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:57.355278 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:57.357188 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:57.357829 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:57.357972 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:57.358390 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:57.358526 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:57.358617 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:57.358822 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:56:57.358904 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5877}
I20260927 22:56:57.359364 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:57.342296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.0.97.61:46355)
0927 22:56:57.342463 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:57.342471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:57.353448 (+ 10977us) server_negotiation.cc:407] Connection header received
0927 22:56:57.353656 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:57.353661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:57.353754 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:57.353848 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:57.354449 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:57.355475 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:57.357308 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:57.357984 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:57.358113 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:57.358619 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:57.358764 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:57.358823 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:57.359022 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 22:56:57.359111 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 22:56:57.363902 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 23182
I20260927 22:56:57.365000 23244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:57.473901 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 23182
I20260927 22:56:57.495013 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 23256
I20260927 22:56:57.496086 23318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:57.606823 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 23256
I20260927 22:56:57.626214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu pid 23330
I20260927 22:56:57.627308 23392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:57.731679 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIqwSmZ/build/asan/bin/kudu with pid 23330
2026-09-27T22:56:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4986 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9282 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (352395 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST